The framework is built on the power of python, meaning any bespoke scripting transformations are easy even for a novice python developer.We were able to migrate a lot of our legacy logic and data warehouse table structure with minimal changes. Clever implementation of Dimension and Fact Table inserts and updates.Lots of good code examples in the documentation.The visualisations are a sort of self-documentation.īut after an initial prototyping sprint, it was very clear that the authors of pygrametl have the right idea. It was always so handy to be able to “see” transformations. There was some level of hesitation in the team about moving towards a scripting language for ETLs as we feared that maintainability would suffer. It’s easy to install and start getting your feet wet. According to the documentation it has been around since 2009 with uptake in healthcare, finance and transport. Research and experimentation lead us to pygrametl – an open source Python ETL framework maintained by a group led by Associate Professor Christian Thomsen from Aalborg University, Denmark. There are, of course, a lot of graphical ETL products in the proprietary software ETL space, some actually quite good, but our experience is that we are for more agile using open source tools that we can experiment and iterate with. They exist, but we didn’t find anything we liked. Initially, we were very keen on staying with a graphical ETL tool, one that was open source of course. It was not quite a “build it again from scratch” scenario, but all were clear that there was some hidden rust with the existing setup. Kettle allowed us to do some great stuff, and the visual interface was helpful for quickly throwing together new transformations.Īs part of a recent review, we decided to assess some of the options that were available to us in order to address some of the shortcomings we were dealing with. Ranging from mammoth perl scripts to graphical interfaces giving visual representations of transformations and data flow.įor some years now, Catalyst had been developing and supporting a particular data warehouse instance, with data imported using Pentaho Kettle (an open source ETL solution). And have yet to let us down.Ĭatalyst has seen and worked with a number of Extract Transform Load (ETL) solutions. Thankfully the tool sets available to us in the open source world are extremely flexible. Whether this be very formal Enterprise system integration engagements, or just custom data migrations. Catalyst has been helping our clients wrestle data from one system to another for many years.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |