Get advice
Arrow White

Integrations
for data exchange between IT systems

Using the ESB approach and Data Lake, we reduce the load on systems, eliminate data loss during exchange, simplify changes in IT systems, and speed up the analysis of all data.

You'll also get

idea

Comparison table of
integrations types

Интеграции
для обмена данными между IT-системами
С помощью ESB подхода и Data Lake снижаем нагрузку на системы, исключаем потерю данных при обмене, упрощаем изменение IT систем, ускоряем анализ всех данных.

Eliminate data loss
when exchanging between systems

Проблема
При обмене “точка-точка” в момент отказа одной из систем данные могут теряться: одна система считает сообщение переданным, когда вторая его не получила.

The problem

During point-to-point exchange, data may be lost at thr moment of failure of one of the systems: one system considers the message transmitted when the second one has not received it.

Решение
Мы настраиваем интеграции так, что при обрыве связи сообщение считается необработанным и будет передано в следующем проходе.

Solution

We configure the integration so that when the connection is broken, the message is considered unprocessed and will be transmitted in the next pass.

Change some IT systems without changing others

Проблема
При старом подходе источник отдает данные, как есть. Потребитель должен преобразовать их под свой формат. При такой схеме любое изменение в источнике повлечет каскад изменений всех связанных потоков данных.

The problem

With the old approach, the source gives the data as it is. The consumer must convert it to his own format. Under this scheme, any change in the source will entail a cascade of changes in all related data streams.

Решение
Изменение ERP или любой другой системы не вызовет доработок всех связанных систем. 
Если система изменится, это затронет только коннекторы “источник-хранилище”. Для всех потребителей все останется без изменения.

Solution

A change in ERP or any other system will not cause improvements to all related systems. 
If the system changes, it's will only affect source-storage connectors. For all consumers, everything will remain unchanged.

Reduce the load on systems without improvements and increasing resources

Проблема
Одни и те же данные нужны разным системам. Потребители дублируют запросы в систему-источник, что увеличивает нагрузку. Если
в компании есть одна система для хранения основных данных, она будет получать огромное число запросов. Часто это старая монолитная система, которую в будущем бизнес хочет заменить.

The problem

Different systems need the same data. Consumers duplicate requests to the source system, which increases the load. If
the company has one system for storing basic data, it will receive a huge number of requests. This is often an old monolithic system that businesses want to replace in the future.

Решение
Потребители обращаются в хранилище. Нагрузка на основной источник снижается без доработок самой системы и увеличения ресурсов. Когда придет время заменить старую систему на несколько новых сервисов, нужно будет изменить потоки “источник-хранилище”. Изменения не затронут множества потребителей.

Solution

Consumers turn to the storage. The load on the main source is reduced without improving the system itself and increasing resources. When the time comes to replace the old system with several new services, you will need to change the source-storage streams. The changes will not affect many consumers.

API as a separate service. Connect consumers of the same type without load

Проблема
Для подключения множества однотипных потребителей (например, розниц) часто создается API внутри системы. При этом нагрузка, которую создают потребители, ложится на эту систему. Недоступность системы-источника станет проблемой для всех потребителей. Они не смогут получить данные.

The problem

To connect multiple consumers of the same type (for example, retailers), an API is often created within the system. At the same time, the burden that consumers create falls on this system. The inaccessibility of the source system will be a problem for all consumers. They won't be able to get data.

Решение
Мы создаем API-коннекторы не внутри сервиса, а отдельным сервисом. Они работают независимо от системы-источника 
и выдерживают высокую нагрузку. Вы сможете обслуживать системы без проблем для множества потребителей.

Solution

We create API connectors not inside the service, but as a separate service.  They operate independently of the source system and withstand high loads. You will be able to maintain systems without problems for many consumers.

We prove using 1C systems as an example: any transformation logic is available

Data transfer
with the “link” type

To transfer an object with the “link” type, you need to make sure that all directories are created in the consumer system. Otherwise, the object will be created broken or the stream will give an error.

Our connectors check whether the system contract is being fulfilled (completeness, correctness). 
Only when all conditions are met, the message will be sent to the consumer

Assembling the object
from several tables

Object information can be stored 
from several sources. You need to request information on each source and combine it for submission.

Our solution allows you to aggregate object data. For example, collect “orders” and “payments” separately (from different sources) and transfer them as “orders with payments”.

Delete a record
from the tabular part of the object

Such a record is not registered for the 1C exchange. The receiver system must know this feature of 1C and calculate changes on its own.

Our connectors can check deletion from the table. If the deletion took place, mark it 
in storage. Consumers don't have to do calculations.

A snapshot of current prices

1C does not provide a snapshot of current prices. The consumer must determine for himself which price from the entire register (past, present, future) is relevant.

Using storage, we calculate current prices so that any consumer system can pick it up without unnecessary logic.

With a ready-made data warehouse
BI implementation is easy

By implementing ESB, we are creating a common structured enterprise data warehouse: Data Warehouse (DWH).

To such a repository it is easy to connect any analytics and report upload system. All data is ready.

We are building the storage architecture in such a way that generating reports on a large amount of information does not reduce the speed of daily data exchange.

Внедряя ESB, мы создаем общее структурированное хранилище данных предприятия: Data Warehouse (DWH).
‍
К такому хранилищу легко подключить любую систему аналитики и выгрузки отчетов. Все данные уже готовы.
‍
Строим архитектуру хранилища так, что формирование отчетов по большому массиву информации не снижает скорость ежедневного обмена данными.

The monitoring system controls the operation of each stream

We log the key stages of each stream. If an error occurs that requires a response, you will receive a message on telegram. It will include a description of the bug and a link to details.

You will be able to respond to incidents proactively, rather than to user requests. The support operator will know exactly where and what went wrong. This will help resolve the incident more quickly.

We will deploy a monitoring system from scratch. Or we will set up monitoring in your infrastructure.

Логируем ключевые этапы работы каждого потока. Если случится ошибка, которая требует реакции, вы получите сообщение в telegram. В нем будет описание ошибки и ссылка на подробности.

Вы сможете реагировать на инциденты проактивно, а не по обращениям пользователей. Оператор техподдержки точно будет знать, где и что пошло не по сценарию. Это поможет решить инцидент быстрее.

Мы развернем систему мониторинга с нуля. Либо настроим мониторинг в вашей инфраструктуре.

The data bus (ESB) is a collection of microservices, not a monolith

With our approach, there is no single point of failure in architecture. A data bus (ESB) is a collection of microservices that are not connected to each other.

ESB components are independent of each other and can run on different servers in different locations.

A failure to exchange either system affects only that system. Everything else is still working.

При нашем подходе в архитектуре нет единой точки отказа. Шина данных (ESB) - это набор микросервисов, не связанных между собой.
‍
Компоненты ESB не зависят друг от друга и могут работать на разных серверах в разных локациях.
‍
Отказ обмена любой из систем затрагивает только эту систему. Все остальное продолжает работать.
Eliminate data loss during exchange
Eliminate data loss during exchange
Change some IT systems without changing others
Reduce the load 
on systems
Create the API as a separate service
Transform data (1C as an example)
Implement BI easily
Control flows with monitoring
Be confident in resiliency
Discuss the task
Risk free model

Get 100%
risk-free results

We are confident in our approach. Tell us about the problem you need to solve. In the documents, we will indicate the result you will receive, the cost and the period (usually 1-2 months).

If you are unhappy with the result after the implementation, you don't have to sign the act and not pay for the work.

We build an architecture of low coupling between IT systems

They trust us

We have implemented a scalable service bus for ATIMO

Learn more
Learn more

We provided Polaris with the ability to easily bring new products to marketplaces

Learn more
Learn more

We have developed a targeted system integration scheme. We implemented ESB technology and launched 48 streams

Learn more
Learn more

Integration of Talend ESB into the IT infrastructure of a logistics company

Learn more
Learn more

We redesigned the IT architecture and drew up a roadmap for its transformation

Learn more
Learn more

We helped to reduce the load and scale system throughput

Learn more
“For some systems, we would not be physically able to digest the load that has increased over the current year if we stayed on the same technologies. We have achieved an effect on a significant and critical parameter for us: we were able to scale bandwidth without scaling sources and receivers.”
Andrey Kurilyuk
Deputy IT Director
Learn more

Work with product data has been automated for the Fix Price network

Learn more
Learn more
No items found.

We will select proven international
open source products

We use Opensource solutions, so our customers reduce license fees without the risk of restrictions from legislation in different countries.

Graphic studio
for creating connectors

A simple and flexible low-code platform that is part of Salesforce with revenues of more than $30 million per year. We're using the Community Edition.

Version control and setup
access rights (roles)

An open source, web-based DevOps lifecycle tool. More than 30 million registered users.

Database
(storage)

A powerful open-source object-relational database system that has been actively developed for over 35 years.

Storage, analysis,
search by logs

Enterprise security, surveillance, and search solutions built on the Elasticsearch platform used by thousands of companies.

Information dashboards
about the state of flows

A system for visualizing and analyzing information that allows you to work “out of the box” with a wide range of data sources.

2
/

If you prefer, we can use other products, including those that require a paid license.

Typical work scheme

We have designed the implementation process so that you get the most out of it. You can go all the way with us. Or order any of the steps separately and hand over the rest to your team.

1. Designing a low coupled architecture

  • We will analyze the current IT architecture, ASIS;
  • We will work out exchanges on key entities;
  • We will design the TOBE architecture;
  • We will prepare a roadmap for the transition to the new architecture;
  • We will prepare recommendations; on tools;
  • We will prepare documentation.
Location Icon
You'll get

The transition plan, taking into account
business features

2. Migrating the streams that are most relevant to you

  • BPMN schemes of flow processes;
  • Deploy and configure the required components
(ETL, storage, logging, monitoring);
  • Configuring connectors;
  • Setting up log collection and monitoring the integration;
  • Documentation and training.
Shield Icon
You'll get

Solving 80% of data exchange problems between systems

3. Migrating the rest of the streams to fill the data warehouse

  • BPMN schemes of flow processes;
  • Configuring connectors;
  • Setting up log collection and monitoring the integration;
  • Documentation and training.
Gear Icon
You'll get

A single mechanism for exchanging the entire enterprise and the completeness of data for analytics

Get a comparison table of integration types

We assessed how three popular types of integrations affect the quality of the IT circuit. The result is presented in a compact comparison table. Submit a request and get the file for free.

Project calculator

Development cost
5,775,000 rubles
The team will develop for
6 weeks
Included in the calculation
Cross or Plus Icon
  • Preparation of a map of systems and data flows (SOA scheme);
  • We will work out exchanges on key entities;
  • Creating connectors for exchanging data for each stream on 3 stands (test, preprod, prod);
  • Setting up to three dashboards per connector within a ready-made monitoring circuit;
  • Documentation on copying integration, reusing, and maintaining;
  • Demonstration of the implemented functionality.
Additional works
Cross or Plus Icon
  • Preparing the infrastructure for connectors to operate;
  • Setting up the monitoring and logging circuit;
  • Creating connectors (storage - receiver) for exchanging data on each high-load stream (>100 messages per minute) on 3 stands (test, preprod, prod);
  • More than 15 attributes for each stream;
  • The cost of computing resources (subscription only);
How many streams will the systems send

Example: The “Product Management System” will send product data. “Order Management System” — about orders. “Warehouse management system” — about the status of the shipment. This is 3 streams.

0
0
100

Example: The “Warehouse Management System” will receive data on goods and orders. “Order Management System” — about goods and shipment status. This is 4 streams.

How many streams will the system receive
0
0
100
Discuss the project

The calculator calculates using an accurate but simplified formula. The scope of work for your project and the final cost may vary. The final calculation will be made by your personal manager.

Discuss the project

The calculator calculates using an accurate but simplified formula. The scope of work for your project and the final cost may vary. The final calculation will be made by your personal manager.

youtube

We are talking about integrations on YouTube-

channel

See all

Integration of Talend ESB into the IT infrastructure of a logistics company

Implementation of the Talend ESB service bus into the project's IT architecture

Integration of the IT infrastructure of an e-commerce project with marketplaces using the WSO2 ESB service bus

Let's discuss your project

Personal managers will contact you

Something went wrong! Please try again.
Something went wrong! Please try again.