Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Dataservices - Data Processing mit Microservices

139 views

Published on

data2day 2018, Heidelberg: Vortrag von Mario-Leander Reimer (@LeanderReimer, Cheftechnologe bei QAware)

=== Dokument bitte herunterladen für bessere Lesbarkeit! ===

Abstract: Data Processing und Microservices sind ein perfektes Gespann. In dieser Kombination können Microservices dazu verwendet werden, ein flexibles, Event-getriebenes und skalierbares System von lose gekoppelten Datenverarbeitungsaufgaben aufzubauen. Diesen Ansatz nennen wir Dataservices.

In diesem Vortrag stellen wir zunächst die wesentlichen Konzepte und einige Schlüsseltechnologien vor, um Dataservice-Architekturen zu realisieren. Anschließend werden wir die einzelnen Bestandteile einer exemplarischen Datenverarbeitungs-Pipeline schrittweise komponieren und die Showcase-Pipeline in der Cloud zur Ausführung bringen und skalieren.

Published in: Data & Analytics
  • Be the first to comment

Dataservices - Data Processing mit Microservices

  1. 1. I1 Pn P1P1 Pn O1
  2. 2. apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx spec: replicas: 3 strategy: type: RollingUpdate template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 livenessProbe: ... readinessProbe: ... --- apiVersion: v1 kind: Service metadata: name: nginx labels: app: nginx namespace: default spec: type: NodePort ports: - port: 80 selector: app: nginx --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx labels: app: nginx namespace: default spec: rules: - host: nginx.cloudkoffer http: paths: - path: / backend: serviceName: nginx servicePort: 80
  3. 3. 33 P1 C1Q1 P1 C1 Q1 Cn P1 C1Q1 CnQn P1 C1 Q1 Q2
  4. 4. CDI Extensions Web Fragments Bean Validation 2.0 CDI 2.0 Managed Beans 1.0 JCA 1.7 JPA 2.2 JMS 2.0 JSP 2.3 EL 3.0 EJB 3.2 Batch 1.0 JSF 2.3 Interceptors 1.2 Mail 1.6 Common Annotations 1.3 JTA 1.2 JAX-WS 1.4 JAX-RS 2.1 Concurrency 1.0 JSON-P 1.1 JSON-B 1.0 WebSocket 1.1 JAPSIC 1.1 JACC 1.5 Security 1.0 Servlet 4.0 JCache 1.0
  5. 5. @MessageDriven(activationConfig = { @ActivationConfigProperty(propertyName = "serverURIs", propertyValue = "tcp://eclipse-mosquitto:1883"), @ActivationConfigProperty(propertyName = "cleanSession", propertyValue = "false"), @ActivationConfigProperty(propertyName = "automaticReconnect", propertyValue = "true"), @ActivationConfigProperty(propertyName = "filePersistence", propertyValue = "false"), @ActivationConfigProperty(propertyName = "connectionTimeout", propertyValue = "30"), @ActivationConfigProperty(propertyName = "maxInflight", propertyValue = "3"), @ActivationConfigProperty(propertyName = "keepAliveInterval", propertyValue = "5"), @ActivationConfigProperty(propertyName = "topicFilter", propertyValue = "de/qaware/oss/cloud/mqtt"), @ActivationConfigProperty(propertyName = "qos", propertyValue = "1") }) public class MqttSourceMDB implements MQTTListener { @OnMQTTMessage @TransactionAttribute(value = TransactionAttributeType.REQUIRED) @Transactional(Transactional.TxType.REQUIRED) public void onMQTTMessage(String topic, MqttMessage message) { JsonReader reader = Json.createReader(new ByteArrayInputStream(message.getPayload())); JsonObject jsonObject = reader.readObject(); // TODO do stuff with the JSON payload } }
  6. 6. JsonObject currentWeather = Json.createObjectBuilder() .add("city", “London") .add("weather", “Drizzle") .build(); StringWriter payload = new StringWriter(); JsonWriter jsonWriter = Json.createWriter(payload); jsonWriter.writeObject(currentWeather); TextMessage msg = session.createTextMessage(payload.toString()); msg.setJMSType("CurrentWeather"); msg.setStringProperty("contentType", "application/vnd.weather.v1+json"); @ActivationConfigProperty(propertyName = "messageSelector", propertyValue = "(JMSType = 'CurrentWeather') AND (contentType = 'application/vnd.weather.v1+json‘)“) JsonReader reader = Json.createReader(new StringReader(body)); JsonObject jsonObject = reader.readObject();

×