beagle.datasources package

Submodules

beagle.datasources.base_datasource module

class beagle.datasources.base_datasource.DataSource[source]

Bases: object

Base DataSource class. This class should be used to create DataSources which are file based.

For non-file based data sources (i.e performing a HTTP request to an API to get some data). The ExternalDataSource class should be subclassed.

Each datasource requires the following annotations be made:

  1. name string: The name of the datasource, this should be human readable.
  2. transformer List[Transformer]: The list of transformers which you can send events from this datasource to.
  3. category string: The category this datasource outputs data to, this should be human readable.

Not supplying these three will not allow the class to get created, and will prevent beagle from loading.

Examples

>>> class MyDataSource(DataSource):
        name = "My Data Source"
        transformers = [GenericTransformer]
        category = "My Category"
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
to_graph(*args, **kwargs) → Any[source]

Allows to hop immediatly from a datasource to a graph.

Supports parameters for the to_graph() function of the transformer.

see :py:method:`beagle.transformers.base_transformer.Transformer.to_graph`

Examples

>>> SysmonEVTX('data/sysmon/autoruns-sysmon.evtx').to_graph(Graphistry, render=True)
Returns:Returns the outuput of the Backends .graph() function.
Return type:Any
to_transformer(transformer: Transformer = None) → Transformer[source]

Allows the data source to be used as a functional API. By default, uses the first transformer in the transformers attribute.

>>> graph = DataSource().to_transformer().to_graph()
Returns:A instance of the transformer class yielded to.
Return type:Transformer
class beagle.datasources.base_datasource.ExternalDataSource[source]

Bases: beagle.datasources.base_datasource.DataSource

This class should be used when fetching data from exteranl sources before processing.

Using a different class allows the web interface to render a different upload page when a data source requiring text input in favor of a file input is used.

Examples

See beagle.datasources.virustotal.generic_vt_sandbox_api.GenericVTSandboxAPI

beagle.datasources.fireeye_ax_report module

class beagle.datasources.fireeye_ax_report.FireEyeAXReport(ax_report: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Yields events one by one from a FireEyeAX Report and sends them to the generic transformer.

The JSON report should look something like this:

{
    "alert": [
        {
            "explanation": {
                "malwareDetected": {
                    ...
                },
                "cncServices": {
                    "cncService": [
                        ...
                },
                "osChanges": [
                    {
                        "process": [...],
                        "registry": [...],
                        ...
                }
            }
        }
    ]
}

Beagle looks at the first alert in the alerts array.

Parameters:ax_report (str) – File path to the JSON AX Report, see class description for expected format.
category = 'FireEye AX'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'FireEye AX Report'
transformers = [<class 'beagle.transformers.fireeye_ax_transformer.FireEyeAXTransformer'>]

beagle.datasources.hx_triage module

class beagle.datasources.hx_triage.HXTriage(triage: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

A FireEye HX Triage DataSource.

Allows generation of graphs from the redline .mans files generated by FireEye HX.

Examples

>>> triage = HXTriage(file_path="/path/to/triage.mans")
category = 'FireEye HX'
events() → Generator[[dict, None], None][source]

Yields each event in the triage from the supported files.

metadata() → dict[source]

Returns basic information about the triage.

  1. Agent ID
  2. Hostname
  3. Platform (win, osx, linux)
  4. Triggering Alert name (if exists)
  5. Link to the controller the triage is from
Returns:Metadata for the submitted HX Triage.
Return type:dict
name = 'FireEye HX Triage'
parse_agent_events(agent_events_file: str) → Generator[[dict, None], None][source]

Generator over the agent events file. Converts each XML into a dictionary. Timestamps are converted to epoch time.

The below XML entry:

<eventItem uid="39265403">
    <timestamp>2018-06-27T21:15:32.678Z</timestamp>
    <eventType>dnsLookupEvent</eventType>
    <details>
    <detail>
        <name>hostname</name>
        <value>github.com</value>
    </detail>
    <detail>
        <name>pid</name>
        <value>12345</value>
    </detail>
    <detail>
        <name>process</name>
        <value>git.exe</value>
    </detail>
    <detail>
        <name>processPath</name>
        <value>c:\windows\</value>
    </detail>
    <detail>
        <name>username</name>
        <value>Bob/Schmob</value>
    </detail>
    </details>
</eventItem>

becomes:

{
    "timestamp": 1530134132,
    "eventType": "dnsLookupEvent",
    "hostname": "github.com",
    "pid": "12345",
    "process": "git.exe",
    "processPath": "c:\windows\",
    "username": "Bob/Schmob",
}
Parameters:agent_events_file (str) – The path to the file containing the agent events.
Returns:Generator over agent events.
Return type:Generator[dict, None, None]
parse_alert_files(temp_dir: str) → Generator[[dict, None], None][source]

Parses out the alert files from the hits.json and threats.json files

Parameters:temp_dir (str) – Folder which contains the expanded triage.
Yields:Generator[dict, None, None] – The next event found in the Triage.
transformers = [<class 'beagle.transformers.fireeye_hx_transformer.FireEyeHXTransformer'>]

beagle.datasources.procmon_csv module

class beagle.datasources.procmon_csv.ProcmonCSV(procmon_csv: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Reads events in one by one from a ProcMon CSV, and parses them into the GenericTransformer

category = 'Procmon'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'Procmon CSV'
transformers = [<class 'beagle.transformers.procmon_transformer.ProcmonTransformer'>]

beagle.datasources.sysmon_evtx module

class beagle.datasources.sysmon_evtx.SysmonEVTX(sysmon_evtx_log_file: str)[source]

Bases: beagle.datasources.win_evtx.WinEVTX

Parses SysmonEVTX files, see beagle.datasources.win_evtx.WinEVTX

category = 'SysMon'
metadata() → dict[source]

Returns the Hostname by inspecting the Computer entry of the first record.

Returns:
>>> {"hostname": str}
Return type:dict
name = 'Sysmon EVTX File'
parse_record(record: lxml.etree.ElementTree, name='') → dict[source]

Parse a single record recursivly into a JSON file with a single level.

Parameters:
  • record (etree.ElementTree) – The current record.
  • name (str, optional) – Last records name. (the default is “”, which [default_description])
Returns:

dict representation of record.
Return type:

dict
transformers = [<class 'beagle.transformers.sysmon_transformer.SysmonTransformer'>]

beagle.datasources.win_evtx module

class beagle.datasources.win_evtx.WinEVTX(evtx_log_file: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Parses Windows .evtx files. Yields events one by one using the python-evtx library.

Parameters:evtx_log_file (str) – The path to the windows evtx file to parse.
category = 'Windows Event Logs'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Get the hostname by inspecting the first record.

Returns:
>>> {"hostname": str}
Return type:dict
name = 'Windows EVTX File'
parse_record(record: lxml.etree.ElementTree, name='') → dict[source]

Recursivly converts a etree.ElementTree record to a JSON dictionary with one level.

Parameters:
  • record (etree.ElementTree) – Current record to parse
  • name (str, optional) – Name of the current key we are at.
Returns:

JSON represntation of the event
Return type:

dict
transformers = [<class 'beagle.transformers.evtx_transformer.WinEVTXTransformer'>]

Module contents

class beagle.datasources.DataSource[source]

Bases: object

Base DataSource class. This class should be used to create DataSources which are file based.

For non-file based data sources (i.e performing a HTTP request to an API to get some data). The ExternalDataSource class should be subclassed.

Each datasource requires the following annotations be made:

  1. name string: The name of the datasource, this should be human readable.
  2. transformer List[Transformer]: The list of transformers which you can send events from this datasource to.
  3. category string: The category this datasource outputs data to, this should be human readable.

Not supplying these three will not allow the class to get created, and will prevent beagle from loading.

Examples

>>> class MyDataSource(DataSource):
        name = "My Data Source"
        transformers = [GenericTransformer]
        category = "My Category"
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
to_graph(*args, **kwargs) → Any[source]

Allows to hop immediatly from a datasource to a graph.

Supports parameters for the to_graph() function of the transformer.

see :py:method:`beagle.transformers.base_transformer.Transformer.to_graph`

Examples

>>> SysmonEVTX('data/sysmon/autoruns-sysmon.evtx').to_graph(Graphistry, render=True)
Returns:Returns the outuput of the Backends .graph() function.
Return type:Any
to_transformer(transformer: Transformer = None) → Transformer[source]

Allows the data source to be used as a functional API. By default, uses the first transformer in the transformers attribute.

>>> graph = DataSource().to_transformer().to_graph()
Returns:A instance of the transformer class yielded to.
Return type:Transformer
class beagle.datasources.SplunkSPLSearch(spl: str, earliest: str = '-24h@h', latest: str = 'now')[source]

Bases: beagle.datasources.base_datasource.ExternalDataSource

Datasource which allows transforming the results of a Splunk search into a graph.

Parameters:spl (str) – The splunk search to transform
Raises:RuntimeError – If there are no Splunk credentials configured.
category = 'Splunk'

Creates a splunk search with query and query_kwargs using splunk_client

Returns:A splunk Job object.
Return type:Job
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
get_results(job, count: int) → list[source]

Return events from a finished Job as an array of dictionaries.

Parameters:job (Job) – Job object to pull results from.
Returns:The results of the search.
Return type:list
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'Splunk SPL Search'
patch_spl(spl: str) → str[source]

Ensures `search ` is the first command in the SPL.

setup_session()[source]
transformers = [<class 'beagle.transformers.generic_transformer.GenericTransformer'>]
class beagle.datasources.CuckooReport(cuckoo_report: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Yields events from a cuckoo sandbox report.

Cuckoo now provides a nice summary for each process under the “generic” summary tab:

{
    "behavior": {
        "generic": [
            {
                'process_path': 'C:\Users\Administrator\AppData\Local\Temp\It6QworVAgY.exe',
                'process_name': 'It6QworVAgY.exe',
                'pid': 2548,
                'ppid': 2460,
                'summary': {
                    "directory_created" : [...],
                    "dll_loaded" : [...],
                    "file_opened" : [...],
                    "regkey_opened" : [...],
                    "file_moved" : [...],
                    "file_deleted" : [...],
                    "file_exists" : [...],
                    "mutex" : [...],
                    "file_failed" : [...],
                    "guid" : [...],
                    "file_read" : [...],
                    "regkey_re" : [...]
                    ...
                },

            }
        ]
    }
}

Using this, we can crawl and extract out all activity for a specific process.

Notes

This is based on the output of the following reporting module: https://github.com/cuckoosandbox/cuckoo/blob/master/cuckoo/processing/platform/windows.py

Parameters:cuckoo_report (str) – The file path to the cuckoo sandbox report.
category = 'Cuckoo Sandbox'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
global_network_events() → Generator[[dict, None], None][source]
identify_processes() → Dict[int, dict][source]

The generic tab contains an array of processes. We can iterate over it to quickly generate Process entries for later. After grabbing all processes, we can walk the “processtree” entry to update them with the command lines.

Returns:
Return type:None
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'Cuckoo Sandbox Report'
process_tree() → Generator[[dict, None], None][source]
transformers = [<class 'beagle.transformers.generic_transformer.GenericTransformer'>]
class beagle.datasources.FireEyeAXReport(ax_report: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Yields events one by one from a FireEyeAX Report and sends them to the generic transformer.

The JSON report should look something like this:

{
    "alert": [
        {
            "explanation": {
                "malwareDetected": {
                    ...
                },
                "cncServices": {
                    "cncService": [
                        ...
                },
                "osChanges": [
                    {
                        "process": [...],
                        "registry": [...],
                        ...
                }
            }
        }
    ]
}

Beagle looks at the first alert in the alerts array.

Parameters:ax_report (str) – File path to the JSON AX Report, see class description for expected format.
category = 'FireEye AX'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'FireEye AX Report'
transformers = [<class 'beagle.transformers.fireeye_ax_transformer.FireEyeAXTransformer'>]
class beagle.datasources.HXTriage(triage: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

A FireEye HX Triage DataSource.

Allows generation of graphs from the redline .mans files generated by FireEye HX.

Examples

>>> triage = HXTriage(file_path="/path/to/triage.mans")
category = 'FireEye HX'
events() → Generator[[dict, None], None][source]

Yields each event in the triage from the supported files.

metadata() → dict[source]

Returns basic information about the triage.

  1. Agent ID
  2. Hostname
  3. Platform (win, osx, linux)
  4. Triggering Alert name (if exists)
  5. Link to the controller the triage is from
Returns:Metadata for the submitted HX Triage.
Return type:dict
name = 'FireEye HX Triage'
parse_agent_events(agent_events_file: str) → Generator[[dict, None], None][source]

Generator over the agent events file. Converts each XML into a dictionary. Timestamps are converted to epoch time.

The below XML entry:

<eventItem uid="39265403">
    <timestamp>2018-06-27T21:15:32.678Z</timestamp>
    <eventType>dnsLookupEvent</eventType>
    <details>
    <detail>
        <name>hostname</name>
        <value>github.com</value>
    </detail>
    <detail>
        <name>pid</name>
        <value>12345</value>
    </detail>
    <detail>
        <name>process</name>
        <value>git.exe</value>
    </detail>
    <detail>
        <name>processPath</name>
        <value>c:\windows\</value>
    </detail>
    <detail>
        <name>username</name>
        <value>Bob/Schmob</value>
    </detail>
    </details>
</eventItem>

becomes:

{
    "timestamp": 1530134132,
    "eventType": "dnsLookupEvent",
    "hostname": "github.com",
    "pid": "12345",
    "process": "git.exe",
    "processPath": "c:\windows\",
    "username": "Bob/Schmob",
}
Parameters:agent_events_file (str) – The path to the file containing the agent events.
Returns:Generator over agent events.
Return type:Generator[dict, None, None]
parse_alert_files(temp_dir: str) → Generator[[dict, None], None][source]

Parses out the alert files from the hits.json and threats.json files

Parameters:temp_dir (str) – Folder which contains the expanded triage.
Yields:Generator[dict, None, None] – The next event found in the Triage.
transformers = [<class 'beagle.transformers.fireeye_hx_transformer.FireEyeHXTransformer'>]
class beagle.datasources.WindowsMemory(memory_image: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Yields events from a raw memory file by leveraging Rekall plugins.

This DataSource converts the outputs of the plugins to the schema provided by GenericTransformer.

Parameters:memory_image (str) – File path to the memory image.
category = 'Windows Memory'
connscan() → Generator[[dict, None], None][source]
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
handles() → Generator[[dict, None], None][source]

Converts the output of the rekall handles plugin to a series of events which represent accessing registry keys or file.

Yields:Generator[dict, None, None] – One file or registry key access event a time.
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'Windows Memory'
pslist() → Generator[[dict, None], None][source]

Converts the output of rekall’s pslist plugin to a series of dictionaries that represent a process getting launched.

Returns:Yields one process launch event
Return type:Generator[dict, None, None]
transformers = [<class 'beagle.transformers.generic_transformer.GenericTransformer'>]
class beagle.datasources.ProcmonCSV(procmon_csv: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Reads events in one by one from a ProcMon CSV, and parses them into the GenericTransformer

category = 'Procmon'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'Procmon CSV'
transformers = [<class 'beagle.transformers.procmon_transformer.ProcmonTransformer'>]
class beagle.datasources.SysmonEVTX(sysmon_evtx_log_file: str)[source]

Bases: beagle.datasources.win_evtx.WinEVTX

Parses SysmonEVTX files, see beagle.datasources.win_evtx.WinEVTX

category = 'SysMon'
metadata() → dict[source]

Returns the Hostname by inspecting the Computer entry of the first record.

Returns:
>>> {"hostname": str}
Return type:dict
name = 'Sysmon EVTX File'
parse_record(record: lxml.etree.ElementTree, name='') → dict[source]

Parse a single record recursivly into a JSON file with a single level.

Parameters:
  • record (etree.ElementTree) – The current record.
  • name (str, optional) – Last records name. (the default is “”, which [default_description])
Returns:

dict representation of record.
Return type:

dict
transformers = [<class 'beagle.transformers.sysmon_transformer.SysmonTransformer'>]
class beagle.datasources.PCAP(pcap_file: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Yields events from a PCAP file.

Parameters:pcap_file (str) – path to a PCAP file.
category = 'PCAP'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'PCAP File'
transformers = [<class 'beagle.transformers.pcap_transformer.PCAPTransformer'>]
class beagle.datasources.GenericVTSandbox(behaviour_report_file: str, hash_metadata_file: str = None)[source]

Bases: beagle.datasources.base_datasource.DataSource

Converts a Virustotal V3 API behavior report to a Beagle graph.

This DataSource outputs data in the schema accepted by GenericTransformer.

Providing the hash’s metadata JSON allows for proper creation of a metadata object. * This can be fetched from https://www.virustotal.com/api/v3/files/{id}

Behavior reports come from https://www.virustotal.com/api/v3/files/{id}/behaviours * Beagle generates one graph per report in the attributes array.

Where {id} is the sha256 of the file.

Parameters:
  • behaviour_report (str) – File containing A single behaviour report from one of the virustotal linked sandboxes.
  • hash_metadata (str) – File containing the hashes metadata, containing its detections.
KNOWN_ATTRIBUTES = ['files_deleted', 'processes_tree', 'files_opened', 'files_written', 'modules_loaded', 'files_attribute_changed', 'files_dropped', 'has_html_report', 'analysis_date', 'sandbox_name', 'http_conversations', 'ip_traffic', 'dns_lookups', 'registry_keys_opened', 'registry_keys_deleted', 'registry_keys_set']
category = 'VT Sandbox'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Generates the metadata based on the provided hash_metadata file.

Returns:Name, number of malicious detections, AV results, and common_name from VT.
Return type:dict
name = 'VirusTotal v3 API Sandbox Report Files'
transformers = [<class 'beagle.transformers.generic_transformer.GenericTransformer'>]
class beagle.datasources.GenericVTSandboxAPI(file_hash: str, sandbox_name: str = None)[source]

Bases: beagle.datasources.base_datasource.ExternalDataSource, beagle.datasources.virustotal.generic_vt_sandbox.GenericVTSandbox

A class which provides an easy way to fetch VT v3 API sandbox data. This can be used to directly pull sandbox data from VT.

Parameters:
  • file_hash (str) – The hash of the file you want to graph.
  • sandbox_name (str, optional) – The name of the sandbox you want to pull from VT (there may be multiple available). (the default is None, which picks the first one)
Raises:

RuntimeError – If there is not virustotal API key defined.

Examples

>>> datasource = GenericVTSandboxAPI(
    file_hash="ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa',
    sandbox_name="Dr.Web vxCube"
)
category = 'VT Sandbox'
name = 'VirusTotal v3 API Sandbox Report'
transformers = [<class 'beagle.transformers.generic_transformer.GenericTransformer'>]
class beagle.datasources.WinEVTX(evtx_log_file: str)[source]

Bases: beagle.datasources.base_datasource.DataSource

Parses Windows .evtx files. Yields events one by one using the python-evtx library.

Parameters:evtx_log_file (str) – The path to the windows evtx file to parse.
category = 'Windows Event Logs'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Get the hostname by inspecting the first record.

Returns:
>>> {"hostname": str}
Return type:dict
name = 'Windows EVTX File'
parse_record(record: lxml.etree.ElementTree, name='') → dict[source]

Recursivly converts a etree.ElementTree record to a JSON dictionary with one level.

Parameters:
  • record (etree.ElementTree) – Current record to parse
  • name (str, optional) – Name of the current key we are at.
Returns:

JSON represntation of the event
Return type:

dict
transformers = [<class 'beagle.transformers.evtx_transformer.WinEVTXTransformer'>]
class beagle.datasources.DARPATCJson(file_path: str)[source]

Bases: beagle.datasources.json_data.JSONFile

category = 'Darpa TC3'
events() → Generator[[dict, None], None][source]

Events are in the format:

“datum”: {
“com.bbn.tc.schema.avro.cdm18.Subject”: {
…

}

This pops out the relevant info under the first key.

name = 'Darpa TC3 JSON'
transformers = [<class 'beagle.transformers.darpa_tc_transformer.DRAPATCTransformer'>]
class beagle.datasources.ElasticSearchQSSerach(index: str = 'logs-*', query: str = '*', earliest: str = '-7d', latest: str = 'now')[source]

Bases: beagle.datasources.base_datasource.ExternalDataSource

Datasource which allows transforming the results of a Elasticsearch Query String search into a graph.

Parameters:
  • index (str) – Elasticsearch index, by default “logs-*”
  • query (str) – Elasticsearch query string, by default “*”
  • earilest (str, optional) – The earliest time modifier, by default “-7d”
  • latest (str, optional) – The latest time modifier, by default “now”
Raises:

RuntimeError – If there are no Elasticsearch credentials configured.
category = 'Elasticsearch'
events() → Generator[[dict, None], None][source]

Generator which must yield each event as a dictionary from the datasource one by one, once the generator is exhausted, this signals the datasource is exhausted.

Returns:Generator over all events from this datasource.
Return type:Generator[dict, None, None]
metadata() → dict[source]

Returns the metadata object for this data source.

Returns:A metadata dictionary to store with the graph.
Return type:dict
name = 'Elasticsearch Query String'
transformers = [<class 'beagle.transformers.generic_transformer.GenericTransformer'>]