Frequently Asked Question

Data Pipes is a cloud-native data management platform that helps organizations quickly become data-driven by creating a shared repository of trusted data, allowing anyone in the company to start analyzing data, while ensuring the data is well-governed. We help simplify and streamline data management for your teams, so you can focus on extracting value from your data.

Data Ingest

With a no code interface, Data Pipes users can create data ingestion pipelines from a variety of data source connectors such as SQL databases, CSV, IoT and REST API. Data Pipes supports Change Data Capture from RDBMS, data is perfectly replicated in minutes, with minimal load on the source systems.

During Ingestion, Data Pipes provides direct integration with cloud hosted data preparation tools for users to clean, normalize and prepare data from an intuitive user interface. In addition, data containing Personally Identifiable Information (PII) is automatically identified and users can select how data containing PII can be masked, tokenized or tagged.

Data Catalog

At the core of Data Pipes is the Data Catalog that provides users with a shared repository of trusted data. Searchable like Google, it allows users to discover and search for existing data, create and edit any metadata from the datasets.

Data Pipes also features an automated Data Lineage that helps the users understand where the data is coming from and how it is being used, to validate that they are working with the right data. With direct integration with cloud hosted analytical tools, Data Pipes allows users to go directly from the catalog to analyzing data, machine learning modeling and sharing them across the organization.

Data Governance

Data Pipes provides a powerful suite of tools to control access to data for users. Authorized users can create rules that grant users access to data based on their Role and Department, these rules can grant or deny access to specific tables, and individual columns and rows.

Individual columns in Data Pipes can be marked as classified, this allows rules to be easily created to deny access for unauthorized users to classified data. Tables containing classified data can also be completely removed from the Catalog. In addition, Data Pipes automatically keeps an audit trail of data access and permission changes by the users in real-time. These audit trails can be easily exported for any audit purposes.

Data Pipes works with expertise partners for custom requirements that may not be in our upcoming roadmap. Speak to one of our product specialists and indicate any special requirements you may need here: datapipes.io/contact

Request for a personalized demo with one of our experts here: datapipes.io/contact

With our expertise working with large enterprises on highly scalable data management platforms and data modeling to meet their needs while adhering to data privacy and security requirements, Data Pipes has seen success in multiple Institutes of High Learnings, Smart Cities and the Public Sector.

Here's one of our recent successes with the SMART Urban Co-Innovation Lab in Singapore

There are three primary users for Data Pipes

Data Office Administrators

Data Office Administrators manage the Data Pipes application. They connect the various data sources to the data lake and organize those datasets in Domains. Data Office Administrators have access to necessary log files to ensure compliance with data security requirements.

Administrators have the ability to create new domains and assign domain owners, create new ingestion pipelines, view access permission rules and export the logs, and manage features for users or a set of users.

Domain Owners

Domain Owners manage access control and data quality for a set of tables. Domain owners typically have some expertise in the IT systems providing the data, which allows them to provide the right context. A typical Domain Owner would be appointed by the head of an operational department (ie. Finance, HR) or be part of a Data Office.

Domain Owners have the ability to document tables to share domain expertise, create access control rules to control who can access their data, and answer to access requests with the option to set a time limit for access.

Data Users

Data Users create data assets used - dashboards, predictive models, etc. Data Pipes provide them with a tightly integrated analytics workbench where they can discover data, validate which dataset to use, request access if necessary, perform data cleaning and preparation, and open the data in typical analytical tools.