This document provides an overview of Apex and Force.com development. It covers Apex language basics, data types, collections, exceptions, asynchronous execution, database integration, triggers, debugging, limits, and unit testing. Key topics include the similarities between Apex and Java, SOQL, DML statements, polymorphism in Apex, and the requirements to deploy code changes to production.
Portable Lucene Index Format & Applications - Andrzej Bialeckilucenerevolution
See conference video - http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011
This talk will present a design and implementation of a flexible, version-independent serialization format for Lucene indexes and its applications in index upgrades / downgrades, in distributed document analysis, in distributed indexing, and in integration with external indexing pipelines. This format enables submitting pre-analyzed documents to Lucene/Solr, and transferring parts of indexes between nodes in a distributed setup.
Large Scale ETL for Hadoop and Cloudera Search using Morphlineswhoschek
Cloudera Morphlines is a new, embeddable, open source Java framework that reduces the time and skills necessary to integrate and build Hadoop applications that extract, transform, and load data into Apache Solr, Apache HBase, HDFS, enterprise data warehouses, analytic online dashboards, or other consumers. If you want to integrate, build, or facilitate streaming or batch transformation pipelines without programming and without MapReduce skills, and get the job done with a minimum amount of fuss and support costs, Morphlines is for you.
In this talk, you'll get an overview of Morphlines internals and explore sample use cases that can be widely applied.
Search is everywhere, and therefore so is Apache Lucene. While providing amazing out-of-the-box defaults, there’s enough projects weird enough to require custom search scoring and ranking. In this talk, I’ll walk through how to use Lucene to implement your custom scoring and search ranking. We’ll see how you can achieve both amazing power (and responsibility) over your search results. We’ll see the flexibility of Lucene’s data structures and explore the pros/cons of custom Lucene scoring vs other methods of improving search relevancy.
Portable Lucene Index Format & Applications - Andrzej Bialeckilucenerevolution
See conference video - http://www.lucidimagination.com/devzone/events/conferences/ApacheLuceneEurocon2011
This talk will present a design and implementation of a flexible, version-independent serialization format for Lucene indexes and its applications in index upgrades / downgrades, in distributed document analysis, in distributed indexing, and in integration with external indexing pipelines. This format enables submitting pre-analyzed documents to Lucene/Solr, and transferring parts of indexes between nodes in a distributed setup.
Large Scale ETL for Hadoop and Cloudera Search using Morphlineswhoschek
Cloudera Morphlines is a new, embeddable, open source Java framework that reduces the time and skills necessary to integrate and build Hadoop applications that extract, transform, and load data into Apache Solr, Apache HBase, HDFS, enterprise data warehouses, analytic online dashboards, or other consumers. If you want to integrate, build, or facilitate streaming or batch transformation pipelines without programming and without MapReduce skills, and get the job done with a minimum amount of fuss and support costs, Morphlines is for you.
In this talk, you'll get an overview of Morphlines internals and explore sample use cases that can be widely applied.
Search is everywhere, and therefore so is Apache Lucene. While providing amazing out-of-the-box defaults, there’s enough projects weird enough to require custom search scoring and ranking. In this talk, I’ll walk through how to use Lucene to implement your custom scoring and search ranking. We’ll see how you can achieve both amazing power (and responsibility) over your search results. We’ll see the flexibility of Lucene’s data structures and explore the pros/cons of custom Lucene scoring vs other methods of improving search relevancy.
ElasticSearch in Production: lessons learnedBeyondTrees
With Proquest Udini, we have created the worlds largest online article store, and aim to be the center for researchers all over the world. We connect to a 700M solr cluster for search, but have recently also implemented a search component with ElasticSearch. We will discuss how we did this, and how we want to use the 30M index for scientific citation recognition. We will highlight lessons learned in integrating ElasticSearch in our virtualized EC2 environments, and challenges aligning with our continuous deployment processes.
Zoe Slattery's slides from PHPNW08:
The ability to store large quantities of local data means that many applications require some form of text search and retrieval facility. From the point of view of the application developer there are a number of choices to make, the first is whether to use a complete packaged solution or whether to use one of the available information libraries to build a custom information retrieval (IR) solution. In this talk I’ll look at the options for PHP programmers who choose to embed IR facilities within their applications.
For Java programmers there is clearly a good range of options for text retrieval libraries, but options for PHP programmers are more limited. At first sight for a PHP programmer wishing to embed indexing and search facilities in their application, the choice seems obvious - the PHP implementation of Lucene (Zend Search Lucene). There is no requirement to support another language, the code is PHP therefore easy for PHP programmers to work with and the license is commercially friendly. However, whilst ease of integration and support are key factors in choice of technology, performance can also be important; the performance of the PHP implementation of Lucene is poor compared to the Java implementation.
In this talk I’ll explain the differences in performance between PHP implementation of Lucene and the Java implementation and examine the other options available to PHP programmers for whom performance is a critical factor.
Faceted search is a powerful technique to let users easily navigate the search results. It can also be used to develop rich user interfaces, which give an analyst quick insights about the documents space. In this session I will introduce the Facets module, how to use it, under-the-hood details as well as optimizations and best practices. I will also describe advanced faceted search capabilities with Lucene Facets.
Cloudera - Using morphlines for on the-fly ETL by Wolfgang HoschekHakka Labs
In this talk Senior Software engineer Wolfgang Hoschek from Cloudera discusses Morphlines, the easy way to build and integrate ETL apps for Hadoop. The talk was recorded at the SumbleUpon offices.
Cloudera Morphlines is a new open source framework that reduces the time and skills necessary to integrate, build, and change Hadoop processing applications that extract, transform, and load data into Apache Solr, Apache HBase, HDFS, enterprise data warehouses, or analytic online dashboards.
Wolfgang Hoschek is a Software Engineer on the Platform team and the lead developer on Morphlines. He is a former CERN fellow and received his Ph.D from the Technical University of Vienna, Austria, and M.S from the University of Linz, Austria.
File Format Benchmarks - Avro, JSON, ORC, & ParquetOwen O'Malley
Hadoop Summit June 2016
The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so. The use cases that we’ve examined are: * reading all of the columns * reading a few of the columns * filtering using a filter predicate * writing the data Furthermore, it is important to benchmark on real data rather than synthetic data. We used the Github logs data available freely from http://githubarchive.org We will make all of the benchmark code open source so that our experiments can be replicated.
Solr Recipes provides quick and easy steps for common use cases with Apache Solr. Bite-sized recipes will be presented for data ingestion, textual analysis, client integration, and each of Solr’s features including faceting, more-like-this, spell checking/suggest, and others.
SoDA v2 - Named Entity Recognition from streaming textSujit Pal
Covers the services supported by SoDA v2. Includes some background on Named Entity Recognition and Resolution, popular approaches to Named Entity Recognition, hybrid approaches, scaling SoDA using Spark and Spark streaming, deployment strategies, etc.
In this talk, Solr's built-in query parsers will be detailed included when and how to use them. Solr has nested query parsing capability, allowing for multiple query parsers to be used to generate a single query. The nested query parsing feature will be described and demonstrated. In many domains, e-commerce in particular, parsing queries often means interpreting which entities (e.g. products, categories, vehicles) the user likely means; this talk will conclude with techniques to achieve richer query interpretation.
Machine learning with Apache Spark MLlib | Big Data Hadoop Spark Tutorial | C...CloudxLab
Big Data with Hadoop & Spark Training: http://bit.ly/2L227PI
This CloudxLab Spark MLlib tutorial helps you to understand Spark MLlib in detail. Below are the topics covered in this tutorial:
1) Introduction to Machine Learning
2) Applications of Machine Learning
3) Machine Learning - Types & Tools
4) Introduction to Spark MLlib
5) Movie Lens Recommendation - Collaborative Filtering in Spark MLlib
Apache Solr is a powerful search and analytics engine with features such as full-text search, faceting, joins, sorting and capable of handling large amounts of data across a large number of servers. However, with all that power and scalability comes complexity. Solr 6 supports a Parallel SQL feature which provides a simplified, well-known interface to your data in Solr, performs key operations such as sorts and shuffling inside Solr for massive speedups, provides best-practices based query optimization and by leveraging the scalability of SolrCloud and a clever implementation, allows you to throw massive amounts of computation power behind analytical queries.
In this talk, we will explore the why, what and how of Parallel SQL and its building block Streaming Expressions in Solr 6 with a hint of the exciting new developments around this feature.
Organizations continue to adopt Solr because of its ability to scale to meet even the most demanding workflows. Recently, LucidWorks has been leading the effort to identify, measure, and expand the limits of Solr. As part of this effort, we've learned a few things along the way that should prove useful for any organization wanting to scale Solr. Attendees will come away with a better understanding of how sharding and replication impact performance. Also, no benchmark is useful without being repeatable; Tim will also cover how to perform similar tests using the Solr-Scale-Toolkit in Amazon EC2.
ORC File and Vectorization - Hadoop Summit 2013Owen O'Malley
Eric Hanson and I gave this presentation at Hadoop Summit 2013:
Hive’s RCFile has been the standard format for storing Hive data for the last 3 years. However, RCFile has limitations because it treats each column as a binary blob without semantics. Hive 0.11 added a new file format named Optimized Row Columnar (ORC) file that uses and retains the type information from the table definition. ORC uses type specific readers and writers that provide light weight compression techniques such as dictionary encoding, bit packing, delta encoding, and run length encoding — resulting in dramatically smaller files. Additionally, ORC can apply generic compression using zlib, LZO, or Snappy on top of the lightweight compression for even smaller files. However, storage savings are only part of the gain. ORC supports projection, which selects subsets of the columns for reading, so that queries reading only one column read only the required bytes. Furthermore, ORC files include light weight indexes that include the minimum and maximum values for each column in each set of 10,000 rows and the entire file. Using pushdown filters from Hive, the file reader can skip entire sets of rows that aren’t important for this query.
Columnar storage formats like ORC reduce I/O and storage use, but it’s just as important to reduce CPU usage. A technical breakthrough called vectorized query execution works nicely with column store formats to do this. Vectorized query execution has proven to give dramatic performance speedups, on the order of 10X to 100X, for structured data processing. We describe how we’re adding vectorized query execution to Hive, coupling it with ORC with a vectorized iterator.
Distributed Model Validation with EpsilonSina Madani
Scalable performance is a major challenge with current model management tools. As the size and complexity of models and model management programs increases and the cost of computing falls, one solution for improving performance of model management programs is to perform computations on multiple computers. The developed prototype demonstrates a low-overhead data-parallel approach for distributed model validation in the context of an OCL-like language. The approach minimises communication costs by exploiting the deterministic structure of programs and can take advantage of multiple cores on each (heterogenous) machine with highly configurable computational granularity. Performance evaluation shows linear improvements with more machines and processor cores, being up to 340x faster than the baseline sequential program with 88 computers.
ElasticSearch in Production: lessons learnedBeyondTrees
With Proquest Udini, we have created the worlds largest online article store, and aim to be the center for researchers all over the world. We connect to a 700M solr cluster for search, but have recently also implemented a search component with ElasticSearch. We will discuss how we did this, and how we want to use the 30M index for scientific citation recognition. We will highlight lessons learned in integrating ElasticSearch in our virtualized EC2 environments, and challenges aligning with our continuous deployment processes.
Zoe Slattery's slides from PHPNW08:
The ability to store large quantities of local data means that many applications require some form of text search and retrieval facility. From the point of view of the application developer there are a number of choices to make, the first is whether to use a complete packaged solution or whether to use one of the available information libraries to build a custom information retrieval (IR) solution. In this talk I’ll look at the options for PHP programmers who choose to embed IR facilities within their applications.
For Java programmers there is clearly a good range of options for text retrieval libraries, but options for PHP programmers are more limited. At first sight for a PHP programmer wishing to embed indexing and search facilities in their application, the choice seems obvious - the PHP implementation of Lucene (Zend Search Lucene). There is no requirement to support another language, the code is PHP therefore easy for PHP programmers to work with and the license is commercially friendly. However, whilst ease of integration and support are key factors in choice of technology, performance can also be important; the performance of the PHP implementation of Lucene is poor compared to the Java implementation.
In this talk I’ll explain the differences in performance between PHP implementation of Lucene and the Java implementation and examine the other options available to PHP programmers for whom performance is a critical factor.
Faceted search is a powerful technique to let users easily navigate the search results. It can also be used to develop rich user interfaces, which give an analyst quick insights about the documents space. In this session I will introduce the Facets module, how to use it, under-the-hood details as well as optimizations and best practices. I will also describe advanced faceted search capabilities with Lucene Facets.
Cloudera - Using morphlines for on the-fly ETL by Wolfgang HoschekHakka Labs
In this talk Senior Software engineer Wolfgang Hoschek from Cloudera discusses Morphlines, the easy way to build and integrate ETL apps for Hadoop. The talk was recorded at the SumbleUpon offices.
Cloudera Morphlines is a new open source framework that reduces the time and skills necessary to integrate, build, and change Hadoop processing applications that extract, transform, and load data into Apache Solr, Apache HBase, HDFS, enterprise data warehouses, or analytic online dashboards.
Wolfgang Hoschek is a Software Engineer on the Platform team and the lead developer on Morphlines. He is a former CERN fellow and received his Ph.D from the Technical University of Vienna, Austria, and M.S from the University of Linz, Austria.
File Format Benchmarks - Avro, JSON, ORC, & ParquetOwen O'Malley
Hadoop Summit June 2016
The landscape for storing your big data is quite complex, with several competing formats and different implementations of each format. Understanding your use of the data is critical for picking the format. Depending on your use case, the different formats perform very differently. Although you can use a hammer to drive a screw, it isn’t fast or easy to do so. The use cases that we’ve examined are: * reading all of the columns * reading a few of the columns * filtering using a filter predicate * writing the data Furthermore, it is important to benchmark on real data rather than synthetic data. We used the Github logs data available freely from http://githubarchive.org We will make all of the benchmark code open source so that our experiments can be replicated.
Solr Recipes provides quick and easy steps for common use cases with Apache Solr. Bite-sized recipes will be presented for data ingestion, textual analysis, client integration, and each of Solr’s features including faceting, more-like-this, spell checking/suggest, and others.
SoDA v2 - Named Entity Recognition from streaming textSujit Pal
Covers the services supported by SoDA v2. Includes some background on Named Entity Recognition and Resolution, popular approaches to Named Entity Recognition, hybrid approaches, scaling SoDA using Spark and Spark streaming, deployment strategies, etc.
In this talk, Solr's built-in query parsers will be detailed included when and how to use them. Solr has nested query parsing capability, allowing for multiple query parsers to be used to generate a single query. The nested query parsing feature will be described and demonstrated. In many domains, e-commerce in particular, parsing queries often means interpreting which entities (e.g. products, categories, vehicles) the user likely means; this talk will conclude with techniques to achieve richer query interpretation.
Machine learning with Apache Spark MLlib | Big Data Hadoop Spark Tutorial | C...CloudxLab
Big Data with Hadoop & Spark Training: http://bit.ly/2L227PI
This CloudxLab Spark MLlib tutorial helps you to understand Spark MLlib in detail. Below are the topics covered in this tutorial:
1) Introduction to Machine Learning
2) Applications of Machine Learning
3) Machine Learning - Types & Tools
4) Introduction to Spark MLlib
5) Movie Lens Recommendation - Collaborative Filtering in Spark MLlib
Apache Solr is a powerful search and analytics engine with features such as full-text search, faceting, joins, sorting and capable of handling large amounts of data across a large number of servers. However, with all that power and scalability comes complexity. Solr 6 supports a Parallel SQL feature which provides a simplified, well-known interface to your data in Solr, performs key operations such as sorts and shuffling inside Solr for massive speedups, provides best-practices based query optimization and by leveraging the scalability of SolrCloud and a clever implementation, allows you to throw massive amounts of computation power behind analytical queries.
In this talk, we will explore the why, what and how of Parallel SQL and its building block Streaming Expressions in Solr 6 with a hint of the exciting new developments around this feature.
Organizations continue to adopt Solr because of its ability to scale to meet even the most demanding workflows. Recently, LucidWorks has been leading the effort to identify, measure, and expand the limits of Solr. As part of this effort, we've learned a few things along the way that should prove useful for any organization wanting to scale Solr. Attendees will come away with a better understanding of how sharding and replication impact performance. Also, no benchmark is useful without being repeatable; Tim will also cover how to perform similar tests using the Solr-Scale-Toolkit in Amazon EC2.
ORC File and Vectorization - Hadoop Summit 2013Owen O'Malley
Eric Hanson and I gave this presentation at Hadoop Summit 2013:
Hive’s RCFile has been the standard format for storing Hive data for the last 3 years. However, RCFile has limitations because it treats each column as a binary blob without semantics. Hive 0.11 added a new file format named Optimized Row Columnar (ORC) file that uses and retains the type information from the table definition. ORC uses type specific readers and writers that provide light weight compression techniques such as dictionary encoding, bit packing, delta encoding, and run length encoding — resulting in dramatically smaller files. Additionally, ORC can apply generic compression using zlib, LZO, or Snappy on top of the lightweight compression for even smaller files. However, storage savings are only part of the gain. ORC supports projection, which selects subsets of the columns for reading, so that queries reading only one column read only the required bytes. Furthermore, ORC files include light weight indexes that include the minimum and maximum values for each column in each set of 10,000 rows and the entire file. Using pushdown filters from Hive, the file reader can skip entire sets of rows that aren’t important for this query.
Columnar storage formats like ORC reduce I/O and storage use, but it’s just as important to reduce CPU usage. A technical breakthrough called vectorized query execution works nicely with column store formats to do this. Vectorized query execution has proven to give dramatic performance speedups, on the order of 10X to 100X, for structured data processing. We describe how we’re adding vectorized query execution to Hive, coupling it with ORC with a vectorized iterator.
Distributed Model Validation with EpsilonSina Madani
Scalable performance is a major challenge with current model management tools. As the size and complexity of models and model management programs increases and the cost of computing falls, one solution for improving performance of model management programs is to perform computations on multiple computers. The developed prototype demonstrates a low-overhead data-parallel approach for distributed model validation in the context of an OCL-like language. The approach minimises communication costs by exploiting the deterministic structure of programs and can take advantage of multiple cores on each (heterogenous) machine with highly configurable computational granularity. Performance evaluation shows linear improvements with more machines and processor cores, being up to 340x faster than the baseline sequential program with 88 computers.
FiloDB: Reactive, Real-Time, In-Memory Time Series at ScaleEvan Chan
My keynote presentation about how we developed FiloDB, a distributed, Prometheus-compatible time series database, productionized it at Apple and scaled it out to handle a huge amount of operational data, based on the stack of Kafka, Cassandra, Scala/Akka.
Global introduction to elastisearch presented at BigData meetup.
Use cases, getting started, Rest CRUD API, Mapping, Search API, Query DSL with queries and filters, Analyzers, Analytics with facets and aggregations, Percolator, High Availability, Clients & Integrations, ...
Use a data parallel approach to process large volumes of data (typically terabytes or petabytes) known as big data.
Focus on reliability and availability of data
CoFX is the framework behind time cockpit (http://www.timecockpit.com). Learn about the data model of CoFX and see how to use it to extend time cockpit.
• Is more intelligent than human ???
– Of course Not!
• A slave that only does what it is told
• Is always right; can never be wrong
• Good at performing lengthy and recurring tasks
Dictionary based Annotation at Scale with Spark, SolrTextTagger and OpenNLPSujit Pal
Presented at Spark Summit EU 2015 at Amsterdam. Details the SoDA project, a micro-service accessible via Spark for naive annotation of large volumes of text against very large lexicons.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
3. Apex Language Basics
• OO, Stored Procedure like language, runs on
Force.com platform only.
• Tightly integrated with Force.com DB.
• Used to code user interfaces and triggers.
• Apex programs run in a multi-tenant
environment => subject to governor limits.
• Integrated testing – code coverage atleast 75%
to be deployed to prod.
4. Declarative Patterns
• Simplicity & Speed
• Data Model – Objects, Fields & Relationships
• Business Logic – Workflow, Validation Rules &
Assignment Rules
• User Interface – Applications, Tabs, Page
Layouts & Record Types
5. Programmatic Patterns via Apex
• Control & Flexibility
• Data Model – Metadata Web Services API.
• Business Logic – VF Controllers, Apex, Web
Services API.
• User Interface – Visualforce Pages, Web
Controls.
6. Overview (contd…)
• Platform automatically upgraded. Force.com will
run all your tests before upgrade.
• No offline runtime environment for Apex
programs, must be deployed to Force.com for
execution.
• Only supported language on Force.com platform.
• Force.com DB is the only DB integrated into Apex.
• 2 choices for development – Web Based App
Builder Tool and Force.com IDE (standalone or
eclipse plug-in).
7. Apex .vs. Java
• Very similar – strongly typed, pass by
reference, Object Oriented, etc.
• Differences include:
• Apex runs in a multi-tenant environment
• Subject to limits
• Case-insensitive
• Transactional behavior is out of the box.
8. Apex Types
• Strongly Typed language, similar to Java.
• String (contains Unicode characters), Boolean,
DateTime, Date, Time, Integer (signed 32-bit),
Long (signed 64-bit), Decimal (signed m.n,
total of 19 characters), Double (signed 64-bit
with decimal), Blob, ID, Object.
• ID stores the unique ID of DB records. Total of
18 characters. Values checked at compile time
& run time. Can throw StringException.
9. sObject Data Type
• Represents a row in the Force.com database.
• Initialized to null by default. Null data type
contains the null constant.
• Declared using the API name for the object.
• Can be instantiated using name/value pairs.
• sObject reference field key can be used to
insert, update or upsert to resolve foreign keys
by external ID.
10. Casting
• Implicit conversion for Numeric Types.
• Integer => Long => Double => Decimal
• ID and String are interchangeable, can throw
StringException.
• Convert from string to numeric types using
valueOf method. Can throw TypeException.
Examples: Integer.valueOf(‘123’),
Decimal.valueOf(‘123.34’)
11. Rounding
• By default, the rounding mode is HALF_EVEN,
which rounds to the nearest neighbor, or to
the even neighbor if equidistant.
• Examples: 0.5 will round to 0, 0.6 will round to
1, 1.5 will round to 2.
12. Date, DateTime, Time
• Use valueOf method on Date and DateTime.
• No valueOf method on Time type.
• For Date, default format is YYYY-MM-DD
• For Datetime: YYYY-MM-DD HH:MM:SS
• DateTime has a format method that follows
the SimpleDateFormat of java.
13. Arrays and Collections
• Collections: List, Set and Map.
• Collections support 4 methods: clear, clone,
isEmpty and size.
• Multi-dimensional arrays are NOT supported.
• 2-D list object by nesting a list within a list is
possible.
• Arrays and Lists are interchangeable. (See
Speaker Notes)
14. Collections
• Not more than 1000 elements in a collection.
• A Collection can contain another collection - not more
than 5 levels deep.
• Cannot add/remove while iterating over a collection.
• Set - unordered collection of unique primitives.
• List - ordered collection of primitives, sObjects,
collections or Apex objects based on indices.
• Map - collection of unique primitive keys, values can be
primitives, sObjects, collections or Apex objects.
15. Exceptions
• Similar to Java using try, catch and finally.
• Common system exceptions:
DmlException (issues with changes to the DB),
NullPointerException,
QueryException (issues with DB queries), and
TypeException (issues converting data types).
• System Exception during recursion – max stack
depth is not documented.
16. Apex Asynchronous Execution
• Asynchronous mode of execution using Future
methods.
• Future methods tagged with @future.
• Used to perform expensive tasks that are not
time-critical.
• A regular synchronous method can begin some
work and invoke a future method to finish it.
• The future method starts fresh with respect to
governor limits (3MB code, 200k l.o.c., 50k
records retrieved from SOQL).
17. Limitations of Future Methods
• Cannot invoke more than 10 future methods in a single
scope of execution.
• FMs cannot call other FMs.
• Signature is always static and void.
• Parameters have to primitive types or collections of
primitive types.
• Cannot test FMs like normal methods (NMs).
• Max of 200 FMs per user within a 24-hour window.
• How to test FMs? Invoke a NM as the only line in a FM and
write test cases for the NM that is invoked in the FM.
• Batch Apex is more powerful than FMs.
18. Database Integration in Apex
• Database integrated into the language & runtime.
• No ORM tools, no DB connection pools.
• Apex code aware of all objects, fields and security
rules.
• All DB Objects (standard and custom) are
implicitly imported as Classes in Apex.
• Use SOQL to retrieve.
• Built-in DML to do CRUD operations.
19. DB Integration (contd…)
• Can write apex code in triggers.
• Apex code runs in a privileged mode, granting full
access to all data. Can be configured to run as a
certain user.
• References to database objects and fields are
verified at compile time.
• Database relationships are also exposed in Apex.
The __r syntax refers to a relationship field, a
field that contains a reference to another object
or list of objects.
20. SOQL Basics
• Centered on a single DB object, specifying 1 or more fields
to retrieve from it.
• Case-insensitive.
• The result of a SOQL query can be a simple list of records
containing rows and columns or hierarchies of records
containing data from multiple, related objects.
• Child-to-Parent SOQL – simple list of child objects. Where
fields from a parent object are included in the query, they
are available as nested variables in Apex code.
• Parent-to-Child SOQL – simple list of parent objects. Each
record of the parent object includes a nested List of child
objects.
21. SOQL For Loop
• Allows it to run when the Project object contains up to
50,000 records for this year without consuming 50,000
records worth of heap space at one time.
Decimal totalHours = 0;
for (Proj__c project : [ SELECT Total_Billable_Hours_Invoiced__c
FROM Proj__c
WHERE Start_Date__c = THIS_YEAR ]) {
totalHours += project.Total_Billable_Hours_Invoiced__c;
}
• Above code is still inefficient & run out of governor limits.
• Change the type of variable in the For loop to a list of
Project records, Force.com provides up to 200 records per
loop iteration. This allows you to modify a whole list of
records in a single operation.
22. DML
• 5 Statements supported – up to 200 records of he
same type.
1. Insert
2. Update (records identified by Id field)
3. Upsert: try update first, otherwise insert.
4. Delete: moves records to Recycle Bin.
5. Undelete: restores records from Recycle Bin.
• Can use DML statements (all or nothing when
there is error) or database methods (allow partial
success).
23. Triggers – when to use them.
• A validation rule is required that is too complex to
define on the database object using formula
expressions.
• Two objects must be kept synchronized. When a
record in one object is updated, a trigger updates
the corresponding record in the other.
• Records of an object must be augmented with
values from another object, a complex calculation
or external data via a Web service call.
24. Triggers
• Work on insert, update, delete and undelete.
• Work with both COs and SOs.
• Before Triggers - update validate values before
saving.
• After Triggers - have access to field values
automatically set like ID or lastUpdated.
• No before undelete event.
• Triggers can be set to active or inactive.
• Specified order of events when a trigger
executes.
25. Trigger Considerations
• All triggers are bulk triggers & can process
multiple records at a time.
• All triggers run as System by default. Override it
by using "with sharing" keywords.
• Triggers cannot contain the static keyword
• Triggers can only contain keywords applicable to
an inner class.
• addError method prevents DML events from
taking action on particular records.
26. Implicit Variables in Triggers
• Trigger.new, Trigger.newMap, Trigger.old,
Trigger.oldMap, isInsert, isUpdate, isExecuting, …
• Trigger.newMap and Trigger.oldMap map IDs to
sObjects.
• Trigger.new is a collection. When used as a bind
variable in a SOQL query, Apex transforms the records
into corresponding ids.
Example:
Contact[] cons = [SELECT LastName FROM Contact
WHERE AccountId IN :Trigger.new];
27. Trigger example
trigger validateTimecard on Timecard__c
(
before insert,
before update
)
{
// Apex code block
}
28. Batch Processing in Triggers
• Always try to bulkify a trigger.
• Bulkifying a trigger – is the process of hardening a trigger
to accept a batch of records.
• triggers can always be invoked with a list of records and
should be optimized accordingly.
• Batches can be up to 200 records.
• Example:
List<Resource__c> toInsert = new List<Resource__c>();
for (Contact contact : contacts) {
toInsert.add(new Resource__c(
Name = contact.FirstName + ' ' + contact.LastName));
}
insert toInsert;
29. Error Handling in Triggers
• Strings describing validation errors can be
added to individual records or fields using the
addError method. Force.com continues to
process the batch, collecting any additional
errors, and then rolls back the transaction and
returns the errors to the initiator of the
transaction.
• All changes are rolled back when there is an
uncaught exception.
30. Database Security in Apex
• Apex always runs in a privileged, system
context that gives it access to read and write
all data.
• Where full access is not appropriate, Apex
provides the with sharing keyword.
31. OO Features
• Quite similar to Java.
• Automatic properties: properties with no
code body. When no code body is present, the
getter and setter logic is implied.
• Example:
public Integer cost { get; set; }
• Access modifiers on automatic properties:
public Integer readOnly { get; private set; }
public Integer writeOnly { private get; set; }
32. Access Modifiers
• Classes: public (visible to the app namespace)
and global (visible to all app namespaces)
• If an inner class is global, its outer class is
required to be global.
• Methods and Properties: private, protected
(class and sub-classes), public and global.
• If a method is global, its class must be global as
well.
• If no access modifier is provided, the method or
variable is private.
33. Polymorphism
• The parent class must specify the virtual or
abstract keywords on the methods to be
overridden.
• In the subclass, the override keyword is used
on the virtual or abstract methods to declare
that they’re replacing implementations in the
parent class.
34. Debug Log
• Cannot step through the code & set breakpoints.
• A debug log records database operations, system processes, and errors that occur
when executing a transaction or while running unit tests.
• The system generates a debug log for a user every time that user executes a
transaction that is included in the filter criteria.
• You can retain and manage the debug logs for specific users.
• Each entry written to the log is assigned a debug log category and level.
• Debug log categories: Database, Apex Code, Apex Profiling, Callout, System,
Validation, Visualforce, Workflow.
• Debug log levels: Error, Warn, Info, Debug, Fine, Finer and Finest. (EWIDFFF)
• To view saved debug logs, click Your Name | Setup | Monitoring | Debug Logs.
35. Limits of Debug Logs
• Once a user is added, that user can record up to 20
debug logs. After a user reaches this limit, debug logs
stop being recorded for that user. Click Reset on the
Monitoring Debug logs page to reset the number of
logs for that user back to 20. Any existing logs are not
overwritten.
• Each debug log can only be 2 MB. Debug logs that are
larger than 2 MB in size are truncated.
• Each organization can retain up to 50 MB of debug
logs. Once your organization has reached 50 MB of
debug logs, the oldest debug logs start being
overwritten.
36. Debug Log Execution & Code Units
• An execution unit is equivalent to a transaction. It
contains everything that occurred within the
transaction. The execution is delimited
by EXECUTION_STARTED and
EXECUTION_FINISHED.
• Code Unit is a discrete unit of work within a
transaction. Units of code are indicated
by CODE_UNIT_STARTED and CODE_UNIT_FINISH
ED. Units of work can embed other units of work.
37. Debug Log Example
EXECUTION_STARTED
CODE_UNIT_STARTED|[EXTERNAL]execute_anonymous_apex
CODE_UNIT_STARTED|[EXTERNAL]MyTrigger on Account trigger event
BeforeInsert for [new]
CODE_UNIT_FINISHED <-- The trigger ends
CODE_UNIT_FINISHED <-- The executeAnonymous ends
EXECUTION_FINISHED
38. Debug Log Line
• Log lines are made up of a set of fields, delimited by a pipe (|).
The format is:
• timestamp: consists of the time when the event occurred and a
value between parentheses. The time is in the user's time zone and
in the format HH:mm:ss.SSS.
• event identifier: consists of the specific event that triggered the
debug log being written to, such
as SAVEPOINT_RESET or VALIDATION_RULE, and any additional
information logged with that event, such as the method name or
the line and character number where the code was executed.
• Example:
11:47:46.038 (38450000)|USER_DEBUG|[2]|DEBUG|Hello World!
39. Debug Log Levels – Execution Path and
Overrides
• When you override the debug log levels for a
class or trigger, these debug levels also apply
to the class methods that your class or trigger
calls and the triggers that get executed as a
result.
• All class methods and triggers in the execution
path inherit the debug log settings from their
caller, unless they have these settings
overridden.
40. View Logs
• In the Force.com IDE, a user interface is provided
for the debug log on the right side of the Execute
Anonymous and Test Runner views. It displays
debug log entries resulting from code executed
by these views.
• Each has a text area for the log entries, a drop-down
list of debug log categories, and a slider for
adjusting the log level.
• Each category receives its own independent log
level, although only one can be shown at a time
in the user interface.
41. Unit Testing
• Test Coverage = 100 x (l.o.c. executed during test runs /
total l.o.c.). Expressed as a percentage. The loc is from
triggers and classes.
• Test Methods – static void methods in an outer class
with the testMethod keyword added.
• Designate a class with the @isTest annotation to treat
all it’s methods as test methods. Classes annotated
with isTest must be declared as private.
• Tests subjected to same governor limits as Apex code.
• Each test method is independently subject to governor
limits and not cumulative with other test methods in a
class.
42. Unit Testing (contd…)
• A test is successful if it executes without encountering
an uncaught exception.
• System.assert( boolean expression ). If the boolean
expression is false, a System.Exception is thrown and
the test fails.
• All database modifications occurring during execution
of a test method are rolled back after the method is
completed.
• Best practices: create own test data in a setup phase
before tests are executed and limit assertions to that
test data. Create environment agnostic test case data.
43. Testing Governor Limits
• Test governor limits using system static
methods provided by Apex - Test.startTest and
Test.stopTest.
• Useful when working with large data sets.
• These static methods allow a test method to
separate the Apex resources and governor
limits being used to prepare and initialize the
dataset from the resources and limits used
during the actual test execution.
44. Test.start & Test.stop Methods
• Test.startTest - marks the point in your test code when your test
actually begins. Each test method is allowed to call this method
only once.
• All of the code before this method should be used to initialize
variables, populate data structures, and so on, allowing you to set
up everything you need in order to run your test.
• After you call this method, you get a fresh set of governor limits for
the remainder of the test until you call Test.stopTest.
• Use Test.stop method in conjunction with the startTest method.
Each test method is allowed to call this method only once. After
calling this method, any post assertions are done in the original
context.
45. System.runAs(user) { …}
• Generally, all Apex code runs in system mode, and the
permissions and record sharing of the current user are
not taken into account.
• System.runAs() lets you write test methods that change
user contexts to either an existing user or a new user.
All of that user's record sharing is then enforced.
• System.runAs() can only be used in a test method.
• The original system context is started again after all
runAs() test methods complete.
• System.runAs() will test and verify proper data sharing
and data access but does not validate CRUD or Field
Level Security permissions.
46. Strategies to test Controllers
• Directly invoke controller getters/setters and
action methods.
• Setup constructor code to read URL
arguments to establish controller state or
perform actions.
• Verify data in the database using SOQL and
SOSL queries.
• Use System.runAs blocks to simulate different
users; System.runAs(user) {block; }.
48. Testing Triggers
• Best Practice for trigger development: Keep
the trigger’s code block as small as possible.
• Place code in a separate class for easier
maintenance and to encourage code reuse.
49. Running Tests
• There are 2 primary ways to execute test methods:
• Salesforce UI
1. Via the Apex Test Execution Page
2. Via the list of classes under Setup > Develop > Apex
Classes
3. Via a specific class under Setup > Develop > Apex
Classes > Class Name
• Metadata API
1. via Force.com IDE
2. runTests() method in the SOAP API
50. Testing – Best Practices
• Strive for 100% code coverage instead of the 75%
requirement.
• Write portable tests that do not rely on data
specific to an environment or org.
• Should not commit to DB nor send emails.
• Use the runAs method to test with different user
contexts.
• Test methods should execute Apex code with
large data sets. Running tests in bulk mode will
test the governor limits of your code.
Editor's Notes
List<Integer> intList = new Integer[3];
intList[0] = 123;
intList.add(456);
Integer[] intArray = new List<Integer>();
intArray.add(456);
intArray.set(0, 123);