Skip to content

Posts

My personal blog containing short random notes that I collect over time on the most varied subjects.

  • in ANN

Achieving Scalable Performance on Commodity Hardware for Large Model Training

Training large-scale artificial intelligence models has become a defining challenge of the modern computational era, often perceived as a domain exclusive to those with access to state-of-the-art, unencumbered supercomputing infrastructure. However, recent advancements have demonstrated that exceptional performance can be achieved even with resource constraints and heterogeneous or restricted hardware. This is accomplished through a sophisticated synthesis of software optimization, algorithmic innovation, and a deep understanding of the underlying system architecture. This post delves into the technical strategies that enable high-throughput training, focusing on innovations in parallelism, communication, and low-precision arithmetic.

  • in PINN

Enhancing Parametric PINN Training: A Synergistic Approach with Weighted Curricula and Learning Rate Scheduling

Physics-Informed Neural Networks (PINNs) have emerged as a powerful paradigm for solving differential equations by embedding physical laws directly into the loss function of a neural network. A particularly compelling application is in solving parametric inverse problems, where the goal is to infer physical parameters (e.g., viscosity, conductivity) from observational data. A common and effective methodology involves a two-stage process: first, training a general parametric model over a range of the parameter, and second, using this pre-trained model as a prior to rapidly identify the specific parameter value that best explains a new set of observations.

  • in PINN

Overcoming Generalization Challenges in 2D Burgers' Equation with Physics-Informed Neural Networks

Physics-Informed Neural Networks (PINNs) have emerged as a powerful paradigm in "scientific machine learning," enabling the integration of physical laws, expressed as Partial Differential Equations (PDEs), into neural network training (Wang et al., 2021, p. 1; Lu et al., 2021, p. 208). While PINNs offer a promising avenue for solving complex, non-linear systems like the 2D Burgers' equation, ensuring robust generalization remains a critical challenge, particularly for solutions exhibiting multi-scale features, anisotropic behavior, or in data-poor environments. The 2D Burgers' equation is a fundamental non-linear PDE often used as a benchmark for such challenges (Raissi et al., 2019, p. 686). This post explores several cutting-edge approaches aimed at enhancing PINN generalization capabilities, directly addressing the underlying limitations that hinder their performance.

  • in PINN

Overcoming Challenges in Physics-Informed Neural Networks (PINNs): Gradient Optimization for Inverse Problems

Physics-Informed Neural Networks (PINNs) represent a promising approach for solving complex Partial Differential Equations (PDEs) and inverse problems, such as determining hidden parameters of a physical system – for instance, viscosity (nu) in a 2D Burgers equation. However, the application of PINNs presents inherent challenges. A recent study by Wang et al. (2020) delves into these limitations and proposes innovative solutions that can significantly enhance PINN performance.

  • in Tools

Marimo

Interesting alternative to Jupyter Notebook, but with other goals and features. The idea is to embed everything inside a single .py file, instead of an .ipynb file. The .py file is used both for the graphical interface (interactive web app) and for command line execution. It tries to eliminate some of Jupyter's reproducibility issues. It has features such as integration with package and project managers, such as uv. It's worth checking out.

https://marimo.io/

Internals

Internals refers to the inner workings of compilers or interpreters, detailing how they are structured and operate, documenting their internal structures such as the low-level runtime library, intermediate representations, control graph analysis and optimization, machine descriptions, and more.

HP-41

The 1979 HP-41 series programmable calculator, produced until 1990, was famously used in the Space Shuttle missions. It featured an LCD alphanumeric display and was powered by a processor initially called "Coconut" (or 1LE3 CPU), later more broadly known as the Nut CPU. The calculator included RAM, ROM, and I/O capabilities, utilizing the interpreted programming language FOCAL ("Forty One Calculator Language"). Both the operating system and the interpreter were stored in ROM, programmed using the Coconut processor's assembly language, also called MCODE (or "M-Code"). A later upgrade, the NEWT Microprocessor, further enhanced its capabilities.

Retrocomputing Archive

The site is a repository dedicated to the preservation and sharing of software and documentation for classical computer systems, including a wide variety of classical computer systems and their software, such as the CP/M operating system. The site hosts software and documentation of all types for classical computer systems, providing a valuable resource for enthusiasts and researchers interested in retrocomputing. Maintained by classiccmp.org community.

http://www.retroarchive.org/