Complex problems require the right expertise! Today one of the biggest challenges is data storage and processing, across these three main domains (3Vs): Volume, Velocity and Variety.
At Xpand IT, the Big Data technological area develops and implements architectures and software solutions that represent the state of the art on the capture, ingestions, process, enrich, storage and management of critical data from huge clusters where the 3Vs are always present. In what concerns to technology stack, we take advantage of almost all state-of-the-art frameworks in Big Data ecosystem such as Spark, Kafka, Hive/Impala, Azure Data Services, MongoDB using Java and Scala as programming language to interact with them.
As a Big Data System Engineer you’ll have a fundamental role in several phases of the adoption of the Big Data platform, participating in the analysis, definition and sizing of distributed storage and / or computing systems, setup, upgrade, securization and tuning. For these critical systems, a special focus on performance and security is crucial, as well as the implementation of the best service development practices to serve as the basis for monitoring tools. Usually this role also works closely with the development teams in the design, development and installation of application solutions, large-scale data processing and storage.
Your daily activities will include:
- Setup / upgrade / securitization / tuning of big data platforms on a large scale in critical environments
- Implement security rules and policies on Big Data platforms
- Recommend and periodically update Big Data platforms (hot fixes, patches, etc.)
- Configure good practices for monitoring the infrastructure of Big Data clusters
- Analyze hardware and software requirements for each project to be implemented
- Design and develop new processes for better stability and performance maintenance of environments
- Participate and help solve performance, scalability and security issues
// Stacks: Shell Linux (e.g., bash), Cloudera, Confluent, Azure Data Services, MongoDB; MIT Kerberos ou Windows Active Directory
SKILLS YOU NEED TO HAVE
- MSc / BSc in Information Systems and Computer Engineering and/or Computer Science
- Good knowledge on Linux operating systems is valued
- Good knowledge of Shell Scripting is valued
- Knowledge of High Availability/Distributed systems and its goals and terminology
- Strong communication skills (written and spoken)
- Team playerand problem solving skills
- Fluent English (written and spoken)
// Will be a nice plus if you have:
// Learn more about Big Data area: