HP Cloud Services conducted performance testing on various VM configurations provided by OpenStack. Benchmark tests were run including byte-unixbench, mbw, iozone, iperf, pgbench and Hadoop wordcount. The results showed the larger VM configurations generally had better performance, but some defects were discovered in 7 out of 20 test VMs, indicating the defect rate was too high for production use. While defects were not directly related to OpenStack, the conclusions were that OpenStack still lacks functionality for production and building a full IaaS service is more complex than the software alone.
Cassandra Compression and Performance EvaluationSchubert Zhang
Even though we had abandoned the Cassandra in all our products, we would like to share our works here.
Why we abandoned the Cassandra in our products? Because:
(1) It is a big wrong in Cassandra's implementation, especially on it's local storage engine layer, i.e. SSTable and Indexing.
(2) It is a big wrong to combine Bigtable and Dynamo. Dynamo's hash ring architecture is a obsolete technolohy for scale, it's consistency and replication policy is also unusable in big data storage.
A unique perspective on what skills are needed for people wanting to work in or make a career of CSR
To keep updated on postings and events go to www.csrtraininginstitute.com and sign up for the newsletter. If interested the CSR Knowledge Centre http://bit.ly/CSRknowledge contains a series of short, pragmatic articles on CSR Strategy, Management and related areas.
Cassandra Compression and Performance EvaluationSchubert Zhang
Even though we had abandoned the Cassandra in all our products, we would like to share our works here.
Why we abandoned the Cassandra in our products? Because:
(1) It is a big wrong in Cassandra's implementation, especially on it's local storage engine layer, i.e. SSTable and Indexing.
(2) It is a big wrong to combine Bigtable and Dynamo. Dynamo's hash ring architecture is a obsolete technolohy for scale, it's consistency and replication policy is also unusable in big data storage.
A unique perspective on what skills are needed for people wanting to work in or make a career of CSR
To keep updated on postings and events go to www.csrtraininginstitute.com and sign up for the newsletter. If interested the CSR Knowledge Centre http://bit.ly/CSRknowledge contains a series of short, pragmatic articles on CSR Strategy, Management and related areas.
Gesù all'umanità- gruppo di Preghiera- Italia http://messaggidivinamisericordia.blogspot.it/
Contatto Mail: gesuallumanitaitalia@yahoo.it
Puoi leggere tutti i messaggi su questo sito: http://illibrodellaverita.blogspot.it/
Data-Intensive Computing for Competent Genetic Algorithms: A Pilot Study us...Xavier Llorà
Data-intensive computing has positioned itself as a valuable programming paradigm to efficiently approach problems requiring processing very large volumes of data. This paper presents a pilot study about how to apply the data-intensive computing paradigm to evolutionary computation algorithms. Two representative cases (selectorecombinative genetic algorithms and estimation of distribution algorithms) are presented, analyzed, and discussed. This study shows that equivalent data-intensive computing evolutionary computation algorithms can be easily developed, providing robust and scalable algorithms for the multicore-computing era. Experimental results show how such algorithms scale with the number of available cores without further modification.
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...Chester Chen
Machine Learning at the Limit
John Canny, UC Berkeley
How fast can machine learning and graph algorithms be? In "roofline" design, every kernel is driven toward the limits imposed by CPU, memory, network etc. This can lead to dramatic improvements: BIDMach is a toolkit for machine learning that uses rooflined design and GPUs to achieve two- to three-orders of magnitude improvements over other toolkits on single machines. These speedups are larger than have been reported for *cluster* systems (e.g. Spark/MLLib, Powergraph) running on hundreds of nodes, and BIDMach with a GPU outperforms these systems for most common machine learning tasks. For algorithms (e.g. graph algorithms) which do require cluster computing, we have developed a rooflined network primitive called "Kylix". We can show that Kylix approaches the rooline limits for sparse Allreduce, and empirically holds the record for distributed Pagerank. Beyond rooflining, we believe there are great opportunities from deep algorithm/hardware codesign. Gibbs Sampling (GS) is a very general tool for inference, but is typically much slower than alternatives. SAME (State Augmentation for Marginal Estimation) is a variation of GS which was developed for marginal parameter estimation. We show that it has high parallelism, and a fast GPU implementation. Using SAME, we developed a GS implementation of Latent Dirichlet Allocation whose running time is 100x faster than other samplers, and within 3x of the fastest symbolic methods. We are extending this approach to general graphical models, an area where there is currently a void of (practically) fast tools. It seems at least plausible that a general-purpose solution based on these techniques can closely approach the performance of custom algorithms.
Bio
John Canny is a professor in computer science at UC Berkeley. He is an ACM dissertation award winner and a Packard Fellow. He is currently a Data Science Senior Fellow in Berkeley's new Institute for Data Science and holds a INRIA (France) International Chair. Since 2002, he has been developing and deploying large-scale behavioral modeling systems. He designed and protyped production systems for Overstock.com, Yahoo, Ebay, Quantcast and Microsoft. He currently works on several applications of data mining for human learning (MOOCs and early language learning), health and well-being, and applications in the sciences.
Gesù all'umanità- gruppo di Preghiera- Italia http://messaggidivinamisericordia.blogspot.it/
Contatto Mail: gesuallumanitaitalia@yahoo.it
Puoi leggere tutti i messaggi su questo sito: http://illibrodellaverita.blogspot.it/
Data-Intensive Computing for Competent Genetic Algorithms: A Pilot Study us...Xavier Llorà
Data-intensive computing has positioned itself as a valuable programming paradigm to efficiently approach problems requiring processing very large volumes of data. This paper presents a pilot study about how to apply the data-intensive computing paradigm to evolutionary computation algorithms. Two representative cases (selectorecombinative genetic algorithms and estimation of distribution algorithms) are presented, analyzed, and discussed. This study shows that equivalent data-intensive computing evolutionary computation algorithms can be easily developed, providing robust and scalable algorithms for the multicore-computing era. Experimental results show how such algorithms scale with the number of available cores without further modification.
SF Big Analytics & SF Machine Learning Meetup: Machine Learning at the Limit ...Chester Chen
Machine Learning at the Limit
John Canny, UC Berkeley
How fast can machine learning and graph algorithms be? In "roofline" design, every kernel is driven toward the limits imposed by CPU, memory, network etc. This can lead to dramatic improvements: BIDMach is a toolkit for machine learning that uses rooflined design and GPUs to achieve two- to three-orders of magnitude improvements over other toolkits on single machines. These speedups are larger than have been reported for *cluster* systems (e.g. Spark/MLLib, Powergraph) running on hundreds of nodes, and BIDMach with a GPU outperforms these systems for most common machine learning tasks. For algorithms (e.g. graph algorithms) which do require cluster computing, we have developed a rooflined network primitive called "Kylix". We can show that Kylix approaches the rooline limits for sparse Allreduce, and empirically holds the record for distributed Pagerank. Beyond rooflining, we believe there are great opportunities from deep algorithm/hardware codesign. Gibbs Sampling (GS) is a very general tool for inference, but is typically much slower than alternatives. SAME (State Augmentation for Marginal Estimation) is a variation of GS which was developed for marginal parameter estimation. We show that it has high parallelism, and a fast GPU implementation. Using SAME, we developed a GS implementation of Latent Dirichlet Allocation whose running time is 100x faster than other samplers, and within 3x of the fastest symbolic methods. We are extending this approach to general graphical models, an area where there is currently a void of (practically) fast tools. It seems at least plausible that a general-purpose solution based on these techniques can closely approach the performance of custom algorithms.
Bio
John Canny is a professor in computer science at UC Berkeley. He is an ACM dissertation award winner and a Packard Fellow. He is currently a Data Science Senior Fellow in Berkeley's new Institute for Data Science and holds a INRIA (France) International Chair. Since 2002, he has been developing and deploying large-scale behavioral modeling systems. He designed and protyped production systems for Overstock.com, Yahoo, Ebay, Quantcast and Microsoft. He currently works on several applications of data mining for human learning (MOOCs and early language learning), health and well-being, and applications in the sciences.
Accelerating HBase with NVMe and Bucket CacheNicolas Poggi
on-Volatile-Memory express (NVMe) standard promises and order of magnitude faster storage than regular SSDs, while at the same time being more economical than regular RAM on TB/$. This talk evaluates the use cases and benefits of NVMe drives for its use in Big Data clusters with HBase and Hadoop HDFS.
First, we benchmark the different drives using system level tools (FIO) to get maximum expected values for each different device type and set expectations. Second, we explore the different options and use cases of HBase storage and benchmark the different setups. And finally, we evaluate the speedups obtained by the NVMe technology for the different Big Data use cases from the YCSB benchmark.
In summary, while the NVMe drives show up to 8x speedup in best case scenarios, testing the cost-efficiency of new device technologies is not straightforward in Big Data, where we need to overcome system level caching to measure the maximum benefits.
Maximizing EC2 and Elastic Block Store Disk Performance (STG302) | AWS re:Inv...Amazon Web Services
Learn tips and techniques that will improve the performance of your applications and databases running on EC2 instance storage and/or Elastic Block Store. This advanced session discusses when to use HI1, HS1, and EBS. It shares an under the hood view on how to tune the performance of Elastic Block Store. The presenter(s) will share best practices on running workloads on EBS such as relational databases (MySQL, Oracle, SQL Server, postgres) and NoSQL data stores such as MongoDB and Riak.
Learn tips and techniques that will improve the performance of your applications and databases running on Amazon EC2 instance storage and/or Amazon Elastic Block Store (EBS). This advanced session discusses when to use HI1, HS1, and Amazon EBS. We will share an "under the hood" view to tune the performance of your Elastic Block Store and best practices for running workloads on Amazon EBS, such as relational databases (MySQL, Oracle, SQL Server, Postgres) and NoSQL data stores, such as MongoDB and Riak.
Accelerating hbase with nvme and bucket cacheDavid Grier
This set of slides describes some initial experiments which we have designed for discovering improvements for performance in Hadoop technologies using NVMe technology
NYJavaSIG - Big Data Microservices w/ SpeedmentSpeedment, Inc.
JAVA MICROSERVICES FOR BIG DATA WITH LOW LATENCY - Per-Ake Minborg, CTO Speedment
By leveraging on memory mapped files (eg. Hazelcast, ChronicleMaps etc.), Speedment supports large Java Maps that easily can exceed the size of your server’s RAM. Because the Java Maps are mapped onto files, these maps can be shared instantly between several microservice JVMs and new microservice instances can be added, removed or restarted very quickly. Data can be retrieved with predictable ultra-low latency for a wide range of operations. The solution can be synchronized with an underlying database so that your in-memory maps will be consistently “alive”. The mapped files can be terabytes which has been done in real world deployment cases and there can be a large number of microservices that shares these maps simultaneously.
In-memory processing has started to become the norm in large scale data handling. This is aclose to the metal analysis of highly important but often neglected aspects of memory accesstimes and how it impacts big data and NoSQL technologies.We cover aspects such as the TLB, the Transparent Huge Pages, the QPI Link, Hyperthreading and the impact of virtualization on high-memory footprint applications. We present benchmarks of various technologies ranging from Cloudera’s Impala to Couchbase and how they are impacted by the underlying hardware.The key takeaway is a better understanding of how to size a cluster, how to choose a cloud provider and an instance type for big data and NoSQL workloads and why not every core or GB of RAM is created equal.
Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss Amazon EBS encryption and share best practices for Amazon EBS snapshot management. Throughout, we share tips for success.
Speakers:
Tom Maddox, AWS Solutions Architect
The lessons I learned is that Open source quickly becomes the natural choice wherever commoditization is happening in the software stack. Thus we expect business-to-business open source, which is already a significant trend in recent history, to become an increasingly common form of open source collaboration. Companies who understand the ground rules of business-to-business open source will be better positioned to identify and take advantage of open source opportunities in the competitive spaces that they share with other companies.
So I will share why open strategy is import for the enterprise. And how to do contributions for the open source projects n today’s topic.
Trystakc.cn was announced in OpenStack Summit San Diego 2012(www.slideshare.net/openstack/trystack-introfinalpdf
).It was a Non-profit OpenStack community projects.
By Stackers, for stackers.Experience the latest OpenStack features.
Welcoming contributions and feedback, Join the fun !
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Enhancing Performance with Globus and the Science DMZGlobus
ESnet has led the way in helping national facilities—and many other institutions in the research community—configure Science DMZs and troubleshoot network issues to maximize data transfer performance. In this talk we will present a summary of approaches and tips for getting the most out of your network infrastructure using Globus Connect Server.
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
2. Introduction
• Virtual Machines
• az-1.region-a.geo-1
• web-created
• Ubuntu 11.04 64 bit
• 3+ VM’s / model
• total 20 VM’s
• Benchmark Suite
• byte-unixbench
• mbw
• iozone
• iperf
• pgbench
• Hadoop wordcount
Mediu XXLar
XSmall Small Large XLarge
m ge
vCPU 1 2 2 4 4 8
• Data Filtering
MEM (GB) 1 2 4 8 16 32
• best VM / model
DISK (GB) 30 60 120 240 480 960
• average by 10
Price ($/hr) 0.04 0.08 0.16 0.32 0.64 1.28
3. byte-unixbench
4500
Si ngl e Thr ead
4000
M t i Thr ead
ul
3500
3000
2500
2000
1500
1000
500
0
XSm l
al Sm l
al M um
edi Lar ge XLar ge XXLar ge
• byte-unixbench index measures overall system performance
• in multi-thread testing, n-Thread = n-vCPU
• system with the same amount of vCPU exhibits similar performance
• memory size does not have much impact on performance
• 2 x vCPU => 1.5 x performance
4. mbw
12000
10000
M CPY
EM
8000 DM
UP
MCBLOCK
6000
4000
2000
0
XSm l
al Sm l
al M um
edi Lar ge XLar ge XXLar ge
• mbw 128
• MB/s
• different systems exhibit similar memory performance
5. iozone – os disk
7000000
w i te
r
6000000 r ew i t e
r
5000000 r andom w i t e
r
r ead
4000000 r er ead
r andom r ead
3000000
2000000
1000000
0
XSm l
al Sm l
al M um
edi Lar ge XLar ge XXLar ge
• iozone -Mcew -i0 -i1 -i2 -s4g -r256k -f /io.tmp
• KB/s
• difference systems exhibit similar write performance
• L / XL / XXL systems exhibit much better read performance
• cgroup blkio throttling? QEMU blk throttle? Different disk types?
6. iozone – data disk
6000000
w i te
r
5000000 r ew i t e
r
r andom w i t e
r
4000000 r ead
r er ead
3000000 r andom r ead
2000000
1000000
0
XSm l
al Sm l
al M um
edi Lar ge XLar ge XXLar ge
• iozone -Mcew -i0 -i1 -i2 -s4g -r256k -f /mnt/io.tmp
• KB/s
• difference systems exhibit similar write performance
• XL / XXL systems exhibit much better read performance
• cgroup blkio throttling? QEMU blk throttle? Different disk types?
7. iperf
XXLarg
XSmall Small Medium Large XLarge
e
XSmall 25 25 25 25 25 25
Small 25 50 50 50 50 50
Medium 25 50 100 100 100 100
Large 25 50 100 200 200 200
XLarge 25 50 100 200 400 400
XXLarge 25 50 100 200 400 650
• (x, y) represents the bandwidth between two systems
• Mbps
• bandwidth limited by the system with lower configuration
• cisco quantum plugin?
8. hadoop wordcount single no
de
900
800 2GB
700
600
500
400
300
200
100
0
XSm l
al Sm l
al M um
edi Lar ge XLar ge XXLar ge
• hadoop wordcount application provided in official distribution
• test directory with 3 files, total file size is 2 GB.
• test result shows the time needed to finish the calculation (s)
9. hadoop wordcount multiple nod
es
1000
900
800
700
600
500
400
300
200
100
0
1 x 2 x 3 x 4 x Sm l
al XXLar ge
Xsm l
al XSm l
al XSm l
al XSm l
al
• dfs.replication = nNodes
• test directory with 3 files, total file size is 2 GB.
• test result shows the time needed to finish the calculation (s)
10. pgbench
1800
1600 Si ngl e Thr ead
M t i Thr ead
ul
1400
1200
1000
800
600
400
200
0
XSm l
al Sm l
al M um
edi Lar ge XLar ge XXLar ge
• postgresql-9.1, postgresql-contrib-9.1
• pgbench -i -s 16 pgbench
• pgbench -t 2000 -c 16 –j n -U postgres pgbench
• in multi-thread testing, n-Thread = n-vCPU
11. defects – pgbench single thr
ead
1200
1000
800 N m
or al
D ect
ef
600
400
200
0
XSm l
al Sm l
al M um
edi Lar ge XLar ge XXLar ge
• defects were observed in all VM models
• test results were smooth on the same VM instance
• the following test results were not affected on defected instances
• mbw
• iperf
• byte-unixbench
12. defects – iozone write result
s
300000
N m
or al
250000 D ect
ef
200000
150000
100000
50000
0
XSm l
al Sm l
al M um
edi Lar ge XLar ge XXLar ge
• test performed on OS disks only
• write performance seems to be the major problem
13. defects – iozone read result
s
7000000
N m
or al
6000000
D ect
ef
5000000
4000000
3000000
2000000
1000000
0
XSm l
al Sm l
al M um
edi Lar ge XLar ge XXLar ge
• test performed on OS disks only
• read performance is similar for all instances in both cases
14. defect rate
7
= %
35
20
• 7 defected instances were found out of 20 total instances
• defect rate too high for deploying production systems
• need extra caution when VM’s are auto-generated by API’s
15. conclusion
先以欲勾牵,后令入佛智。
鸠摩罗什大师译 《维摩诘所说经 . 佛道品第八 》
• HP defects were not directly related to OpenStack
• OpenStack still lacks key functionalities for production deployment
• building IaaS service is more complicated than installing OpenStack
• open source IaaS software => IaaS support and service => $$$