Put the ModSecurity interception logic onto separated threads

[wfjsw]

This PR is created to describe my patch to fix #227 and it is by no means a complete patch ready for merge.

The patch contains several unrelated changes, namely:

• Logger change, from dd to ngx_log_error to accomodate my own debugging need
• Removal of ngx_http_modsecurity_pcre_malloc_init and ngx_http_modsecurity_pcre_malloc_done. They are not used in my configuration where PCRE2 is used, and it looks suspicious for SEGVs so I commented them out as a precaution.
• Why use NGX_OK in logging handler? Changed to NGX_DECLINED.

Not yet implemented:

• I'm not sure whether logging phase would take advantage of this. I did not change that yet.
• Missing a NGX_THREADS guard for Nginx setup without threading support.

Currently it passes all test suites and performs well in production.

Benchmarking is welcomed.

[airween]

Hi @wfjsw, there is a new CI workflow test in this repository. Could you pick up the modifications to enable run those tests? Thanks!

[jeremyjpj0916]

@airween @wfjsw I wonder if multi threading is at least the partial answer to fix the long known problem of ModSecurity+nginx being a performance killer for throughput.

[wfjsw]

I no longer have a ModSecurity install on my machine so I'm unable to investigate further :(

Re performance issue: From the CPU loads it still seems heavy. I'd say there is real computing constraint in WAF. (Or it might be a problem caused by PCRE2. Who knows)

[SonNgo2211]

Hi @airween,
I have read this blog post: https://blog.litespeedtech.com/2021/02/22/asynchronous-modsecurity-engine/
It mentions the asynchronous ModSecurity engine and how request blocking is handled.
Has this topic been included in the roadmap or is there any related plan for further enhancements?
Thanks!

[airween]

Hi @SonNgo2211,

Hi @airween, I have read this blog post: https://blog.litespeedtech.com/2021/02/22/asynchronous-modsecurity-engine/ It mentions the asynchronous ModSecurity engine and how request blocking is handled. Has this topic been included in the roadmap or is there any related plan for further enhancements? Thanks!

Thanks for this link. I'm not a LiteSpeed expert (I haven used that at all yet), but I think the key is this:
"As of LiteSpeed Web Server v6.0, we have introduced a separate worker thread pool for ModSecurity processing."

As I understand that the solution is based on a web server (LiteSpeed) function. So there is no similar function in Nginx, I'm afraid there are no plans to use this mechanism in Nginx.

[SonNgo2211]

I noticed that Nginx supports asynchronous thread pools for offloading blocking main working cycle (see: https://www.nginx.com/blog/thread-pools-boost-performance-9x/).

As far as I know, ModSecurity/ModSecurity-Nginx connector run synchronously within the Nginx worker’s event loop. This means heavy or complex rules can still block the worker, and the Nginx thread pools cannot currently be used to offload WAF evaluation away from the main event loop.

This PR discussing separated threads for ModSecurity (similar solution of LiteSpeed). Is there any technical constraint or design decision that prevents such an approach in connector Modsecurity-Nginx? I’d love to learn more. Thanks @airween!

[SonNgo2211]

I noticed that Nginx supports asynchronous thread pools for offloading blocking main working cycle (see: https://www.nginx.com/blog/thread-pools-boost-performance-9x/).

Sorry for my initial confusion!
I read the blog more thoroughly and did some more digging. It seems that Nginx thread pools are only used to offload blocking disk I/O operations (such as file reads/writes and sendfile operations).

I would still be interested to learn if there are any future plans/discussions about supporting asynchronous/multi-threaded WAF assessments in the ModSecurity-nginx connector. I will continue to monitor the issue owasp-modsecurity/ModSecurity#3215

[airween]

I would still be interested to learn if there are any future plans/discussions about supporting asynchronous/multi-threaded WAF assessments in the ModSecurity-nginx connector.

Thanks again for bringing up this topic! Actually there is no any plan/discussion about async/mt threaded developing, but this does not mean we don't want to discuss it.

Currently I'm working on the libmodsecurity3's API changes (it's necessary to change the current API), then I have to align the connector for it. Probably the connector version will be 1.1.0 then.

After that, we can start to think about this topic.

[SonNgo2211]

Thanks a lot @airween!
It's great to see that the topic of asynchronous/multithreaded WAF is at least still open for future discussion. I hope to have a lot of fun with the upcoming API release in libmodsecurity3! As a Modsecurity user I'm happy to contribute use cases and issues from real implementations to help it grow.