This writeup is assuming you are tweaking to the minimise swap and maximise physical memory. This tweaking should be considered especially for High Performance MPI applications where good low latency parallelism between nodes is very essential.
In addition, this writeup also help you to “kill” runaway memory applications. For more information, see Tweaking the Linux Kernel to manage memory and swap (Linux Cluster)
Tuesday, August 30, 2011
Monday, August 29, 2011
Network File System (NFS) in High Performance Network
This article "High Performance (NFS) in High Performance" by Carnegic Mellon is very interesting article about NFS Performance. Do take a look. Here is a summary of their fundings
- For point-to-point throughput, IP over InfiniBand (Connected Mode) is comparable to a native InfiniBand.
- When a disk is a bottleneck, NFS can benefit from neither IPoIB nor RMDA
- When a disk is not a bottleneck, NFS benefits significantly from both IPoIB and RDMA. RDMA is better than IPoIB by ~20%
- As the number of concurrent read operations increases, aggregate throughputs achieved for both IPoIB and RDMA significantly improve with no disadvantage for IPoIB
Sunday, August 28, 2011
High NFS Load causing echo 0 > /proc/sys/kernel/hung_task_timeout_secs
Do note that simultaneous numerous write by the NFS Clients on the NFS Server will cause tremendous performance penalty and system lock-out as describe below. You will notice if you use "top" utilities, the load can be extremely high as numerous system locks are queued.
One of my researcher was running a intense load on the NFS Server that cause an eventual "echo 0 > /proc/sys/kernel/hung_task_timeout_secs". Before that, I saw on the log file "rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket"
To solve the problem, you have to lighten the load of the NFS or improve the setting. You may want to take a look at the Configuring NFS Server for Performance. A longer term solution will be to move to parallel file system.
Sometimes, it could be caused by other factors like drivers. You may want to take a look at Upgrading of Broadcom Drivers to resolve eth0 NIC SerDES Link is Down
One of my researcher was running a intense load on the NFS Server that cause an eventual "echo 0 > /proc/sys/kernel/hung_task_timeout_secs". Before that, I saw on the log file "rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket"
To solve the problem, you have to lighten the load of the NFS or improve the setting. You may want to take a look at the Configuring NFS Server for Performance. A longer term solution will be to move to parallel file system.
Sometimes, it could be caused by other factors like drivers. You may want to take a look at Upgrading of Broadcom Drivers to resolve eth0 NIC SerDES Link is Down
Saturday, August 27, 2011
'Devastating' Apache bug leaves servers exposed
'Devastating' Apache bug leaves servers exposed
Maintainers of the Apache webserver are racing to patch a severe weakness that allows an attacker to use a single PC to completely crash a system and was first diagnosed 54 months ago.
Attack code dubbed “Apache Killer” that exploits the vulnerability in the way Apache handles HTTP-based range requests was published Friday on the Full-disclosure mailing list. By sending servers running versions 1.3 and 2 of Apache multiple GET requests containing overlapping byte ranges, an attacker can consume all memory on a target system.
The denial-of-service attack works by abusing the routine web clients use to download only certain parts, or byte ranges, of an HTTP document from an Apache server. By stacking an HTTP header with multiple ranges, an attacker can easily cause a system to malfunction. On Wednesday morning, Apache developers said they expect to release a patch in the next 96 hours.
The Apache advisory contains several workarounds that admins can deploy in the meantime.
The susceptibility of Apache's range handling to crippling DoS attacks was disclosed in January 2007 Michal Zalewski, a security researcher who has since taken a job with Google. He said at the time that both Apache and Microsoft's competing IIS webserver were vulnerable to crippling DoS attacks because of the programs' “bizarro implementation” of range header functionality based on the HTTP/1.1 standard.
“Combined with the functionality of window scaling (as per RFC 1323)), it is my impression that a lone, short request can be used to trick the server into firing gigabytes of bogus data into the void, regardless of the server file size, connection count, or keep-alive request number limits implemented by the administrator,” Zalewski wrote. “Whoops?”
In an email to The Register on Wednesday, Zalewski wrote: “Not sure why they haven't done something about it back then, probably just haven't noticed in absence of an exploit.”
The episode challenges the conventional wisdom repeated by many proponents of open-source software that flaws in freely available software get fixed faster than in proprietary code because everyday users are free to inspect the source code and report any vulnerabilities they find. Assuming that claim is true, the four-year weakness in Apache's range-handling feature would appear to be an obvious exception.
About 235 million websites use Apache, making it the most widely used webserver with about 66 percent of the entire internet, according to figures released last month by Netcraft. IIS ranked second with more than 60 million sites, or about 17 percent.
In a statement issued several hours after this article was published, Microsoft spokesman Jerry Bryant said: "IIS 6.0 and later versions are not susceptible to this type of denial-of-service due to built in restrictions." ®
Update
Devs race to fix weakness disclosed in 2007
Posted in Security, 24th August 2011 18:05 GMT
Maintainers of the Apache webserver are racing to patch a severe weakness that allows an attacker to use a single PC to completely crash a system and was first diagnosed 54 months ago.
Attack code dubbed “Apache Killer” that exploits the vulnerability in the way Apache handles HTTP-based range requests was published Friday on the Full-disclosure mailing list. By sending servers running versions 1.3 and 2 of Apache multiple GET requests containing overlapping byte ranges, an attacker can consume all memory on a target system.
“The behaviour when compressing the streams is devastating and can end up in rendering the underlying operating system unusable when the requests are sent parallely,” Kingcope, the researcher credited with writing and publishing the proof-of-concept attack code wrote Wednesday on Apache's Bugzilla discussion list. “Symptoms are swapping to disk and killing of processes including but solely httpd processes.”
The denial-of-service attack works by abusing the routine web clients use to download only certain parts, or byte ranges, of an HTTP document from an Apache server. By stacking an HTTP header with multiple ranges, an attacker can easily cause a system to malfunction. On Wednesday morning, Apache developers said they expect to release a patch in the next 96 hours.
The Apache advisory contains several workarounds that admins can deploy in the meantime.
The susceptibility of Apache's range handling to crippling DoS attacks was disclosed in January 2007 Michal Zalewski, a security researcher who has since taken a job with Google. He said at the time that both Apache and Microsoft's competing IIS webserver were vulnerable to crippling DoS attacks because of the programs' “bizarro implementation” of range header functionality based on the HTTP/1.1 standard.
“Combined with the functionality of window scaling (as per RFC 1323)), it is my impression that a lone, short request can be used to trick the server into firing gigabytes of bogus data into the void, regardless of the server file size, connection count, or keep-alive request number limits implemented by the administrator,” Zalewski wrote. “Whoops?”
In an email to The Register on Wednesday, Zalewski wrote: “Not sure why they haven't done something about it back then, probably just haven't noticed in absence of an exploit.”
The episode challenges the conventional wisdom repeated by many proponents of open-source software that flaws in freely available software get fixed faster than in proprietary code because everyday users are free to inspect the source code and report any vulnerabilities they find. Assuming that claim is true, the four-year weakness in Apache's range-handling feature would appear to be an obvious exception.
About 235 million websites use Apache, making it the most widely used webserver with about 66 percent of the entire internet, according to figures released last month by Netcraft. IIS ranked second with more than 60 million sites, or about 17 percent.
In a statement issued several hours after this article was published, Microsoft spokesman Jerry Bryant said: "IIS 6.0 and later versions are not susceptible to this type of denial-of-service due to built in restrictions." ®
Update
Trustwave's SpiderLabs has provided a detailed technical analysis here along with instructions for mitigating attacks using the open-source ModSecurity firewall.
Friday, August 26, 2011
Infiniband HOWTo
Stumble on this short but wonderful tutorial on Infiniband HOWTo by Guy Coates. Although the article is for Debian, but you can apply to CentPS
Subscribe to:
Posts (Atom)
