Sign up

Streamlining Boomi Container Logs with Parseable

Streamlining Boomi Container Logs with Parseable

Nikhil Sinha's photo
Nikhil Sinha
Jun 25, 2024·

5 min read

Table of contents

Boomi is a leading integration Platform as a Service (iPaaS) that allows you to design, deploy, and manage integration processes. However, managing logs can be challenging. Using Parseable with Boomi reduces the headache and enables the user to experience a seamless process.

Boomi logs are a goldmine of information, helping users get the information they need, yet there are significant challenges in accessing and interpreting the logs.

Traditionally, these logs are stored in files, and extracting meaningful insights often involves manual efforts that are both time-consuming and error-prone.

In this article, learn how to use Parseable with Boomi to utilize its full potential.

Understanding Boomi runtimes and logs

Boomi Runtimes, such as Atoms, Molecules, and Clouds, are the engines that execute integration processes. These runtimes generate several types of logs, each serving a specific purpose in monitoring and troubleshooting.

While Boomi offers some built-in tools, they often need to catch up, leaving you craving more insights. By integrating Boomi container logs with Parseable, you can streamline the process of monitoring and troubleshooting your Boomi runtimes.

Types of Boomi logs

Boomi generates several types of logs, each crucial in maintaining and troubleshooting your integrations. Let's break down the key types:

  • Process Logs: Process logs capture details about the execution of processes, such as steps executed, data processed, and any errors encountered. Process logs are helpful for ensuring integrations run smoothly and for debugging.

  • Container Logs: Container logs provide insight into the overall health and activities of the runtime environment. They include information such as Atom status (startup, shutdown, restart), unexpected runtime errors, listener server startup/shutdown, processing of MDN messages, and communication with the AtomSphere platform.

  • Web Server Logs: Web server logs record HTTP requests and responses for web services hosted on Boomi, helping to monitor and troubleshoot API interactions.

Challenges with Boomi Logs

There are significant challenges in accessing and interpreting Boomi logs. For example:

  • Volume and Complexity: Boomi generates significant log data, especially in environments with numerous integration processes. Sifting through these logs to find relevant information can be overwhelming.

  • Real-Time Monitoring: Modern business challenges need real-time monitoring and alerts, which Boomi does not offer.

  • Unfriendly UI: Traditional log file formats are not designed for easy readability. Developers and administrators often have to deal with cryptic messages and raw data, making it challenging to diagnose problems quickly.

  • Performance Impact: Constantly accessing and parsing large log files can impact system performance. This trade-off between maintaining observability and ensuring system efficiency is a common pain point.

Ingesting Boomi Logs into Parseable using Vector

Vector, a log agent, can efficiently handle large log files and transform them into structured data. Here's a step-by-step guide to ingesting Boomi container logs into Parseable.

Download the log from the Boomi atmosphere UI

Navigate to Manage > Atom Management > select Atom > Logs > Atom Logs. This is how a sample Boomi Container Log File looks:

Dec 21, 2023 12:00:00 AM UTC INFO
[com.boomi.process.ProcessListener%ProcessRunner run] Skipping execution of process Main:PDM9000ToMarxLogicPartsDataProc [alstomtransport-BFJG449], schedules are paused.
Dec 21, 2023 12:00:00 AM UTC INFO
[com.boomi.process.ProcessListener%ProcessRunner run] Skipping execution of process Main:ALPSToSNOWJobApplicationDataProc [alstomtransport-BFJG449], schedules are paused.
Dec 21, 2023 12:00:01 AM UTC INFO
[com.boomi.process.ProcessListener%ProcessRunner run] Skipping execution of process Main:SOLToOrchestraExternalSupplierProc [alstomtransport-BFJG449], schedules are paused.
Dec 21, 2023 12:00:02 AM UTC INFO
[com.boomi.connector.databaseconnector.get.StandardGetOperation prepareStatement] Values appended for prepared statement
Dec 21, 2023 12:00:14 AM UTC WARNING
[com.boomi.process.shape.NotifyShape appendMessage] Notify Shape message size exceeded.

Configure Vector to Read the Log File

Set up Vector to read the container log file from your system. This involves specifying the file path, delimiter, and other relevant parameters in the Vector configuration file.

api:
  enabled: false
sources:
  in:
    type: file
    include:
      - "/Users/nikhilsinha/Vector/container.log"
    data_dir: "/Users/nikhilsinha/Vector"
    line_delimiter: "\n"
    max_line_bytes: 102400
    max_read_bytes: 2048
    offset_key: offset
    read_from: beginning
    rotate_wait_secs: 92233720

Transform the Log Data

Use Vector Remap Language (VRL) to parse and transform the log data into meaningful fields.

transforms:
  my_transform:
    inputs:
      - "in"
    type: "remap"
    source: >-
      . |= parse_regex!(.message, r'^(?P<log_timestamp>[A-Za-z]{3} \d{1,2}, \d{4} \d{2}:\d{2}:\d{2} [APMapm]{2})\s(?P<timezone>\w+)\s(?P<loglevel>\w+)\s+(?P<message>.*)$')

Send Transformed Data to Parseable

Configure Vector to send the transformed log data to Parseable using HTTP.

sinks:
  parseable:
    type: http
    method: post
    batch:
      max_bytes: 10485760
      max_events: 1000
      timeout_secs: 10
    compression: gzip
    inputs: 
      - my_transform
    encoding:
      codec: json
    uri: http://localhost:8000/api/v1/ingest
    auth:
      strategy: basic
      user: admin
      password: admin
    request:
      headers:
        X-P-Stream: vectordemo2
    healthcheck:
      enabled: true
      path: http://localhost:8000/api/v1/liveness
      port: 8010

Benefits of Using Parseable

Once ingested into Parseable, the log data becomes much easier to query and analyze. To quickly identify and address issues, you can filter logs by various criteria such as log level, timestamp, or specific keywords. This streamlined approach significantly reduces the time and effort required to manage and troubleshoot Boomi runtimes.

Example of Transformed Log Event

Here is an example of a log event after ingestion into Parseable:

{
  "file": "/Users/nikhilsinha/Vector/container.log",
  "host": "NikhilSinha.local",
  "loglevel": "WARNING",
  "message": "[com.boomi.process.shape.NotifyShape appendMessage] Notify Shape message size exceeded.",
  "offset": 8948,
  "p_metadata":" ",
  "p_tags":" ",
  "p_timestamp": "2024-06-24T18:03:34.502",
  "source_type": "file",
  "timestamp": "Dec 21 2023 12:00:14 AM",
  "timezone": "UTC"
}

Create-Stream

Once ingested, it becomes simpler to query and analyze the data in Parseable. For example, you can filter by log level to find the error logs.

Filter-Stream

In this image, you can see that 650k lines from the log file were ingested to Parseable in less than a minute.

Stats-Stream

With this structured data, you can efficiently monitor and analyze the health of your Boomi runtimes, ensuring that any issues are promptly identified and resolved.

Wrapping Up

Boomi is a robust integration platform, but managing its extensive log files can be challenging. By leveraging Vector and Parseable, you can transform raw log data into meaningful insights, making it easier to maintain the health and efficiency of your Boomi runtimes. This integration not only simplifies log management but also enhances your ability to detect and troubleshoot issues quickly, ultimately improving the reliability and performance of your integration processes.

To learn more, visit our pricing page.
To get started or try Parseable, visit our demo page.

BoomiParseableintegrations