Porting Phi-3-Vision to MLX: A Python Hobbyist’s Journey

Hey fellow devs! đź‘‹

I’ve just published a comprehensive series on Medium detailing my journey of porting Phi-3-Vision, a powerful vision-language model, from Hugging Face to Apple’s MLX framework. As a Python hobbyist, I wanted to share my experience a…


This content originally appeared on DEV Community and was authored by Josef Albers

Hey fellow devs! đź‘‹

I've just published a comprehensive series on Medium detailing my journey of porting Phi-3-Vision, a powerful vision-language model, from Hugging Face to Apple's MLX framework. As a Python hobbyist, I wanted to share my experience and hopefully inspire others to dive into AI model optimization.

đź“š Series Overview:

  1. Basic Implementation: Getting Phi-3-Vision up and running in MLX.
  2. Su-scaled Rotary Position Embeddings (SuRoPE): Implementing 128K context support.
  3. Batching: Optimizing for multiple inputs.
  4. Caching: Speeding up text generation.
  5. Choice Selection: Implementing constrained output.
  6. Constrained Decoding: Guiding the model's output structure.
  7. LoRA Training: Fine-tuning the model efficiently.
  8. Agent and Toolchain System: Building flexible AI workflows.

🤔 Why This Matters:

  • Run advanced AI models efficiently on Apple Silicon
  • Learn about model optimization techniques
  • Understand the internals of vision-language models
  • Explore the capabilities of MLX for AI development

đź”— Read the Full Series:

https://medium.com/@albersj66

đź’» GitHub Repository:

I've open-sourced all the code and markdown files used in this series. You can find them in my GitHub repository:

https://github.com/JosefAlbers/Phi-3-Vision-MLX

Feel free to explore, experiment, and contribute!

đź’¬ Let's Discuss:

  • Have you worked with MLX or other AI frameworks on Apple Silicon?
  • What challenges have you faced in porting or optimizing AI models?
  • Any specific parts of the series you'd like to dive deeper into?

I'm excited to hear your thoughts and experiences! Let's learn from each other and push the boundaries of what's possible with AI on consumer hardware.


This content originally appeared on DEV Community and was authored by Josef Albers


Print Share Comment Cite Upload Translate Updates
APA

Josef Albers | Sciencx (2024-08-07T01:38:29+00:00) Porting Phi-3-Vision to MLX: A Python Hobbyist’s Journey. Retrieved from https://www.scien.cx/2024/08/07/porting-phi-3-vision-to-mlx-a-python-hobbyists-journey/

MLA
" » Porting Phi-3-Vision to MLX: A Python Hobbyist’s Journey." Josef Albers | Sciencx - Wednesday August 7, 2024, https://www.scien.cx/2024/08/07/porting-phi-3-vision-to-mlx-a-python-hobbyists-journey/
HARVARD
Josef Albers | Sciencx Wednesday August 7, 2024 » Porting Phi-3-Vision to MLX: A Python Hobbyist’s Journey., viewed ,<https://www.scien.cx/2024/08/07/porting-phi-3-vision-to-mlx-a-python-hobbyists-journey/>
VANCOUVER
Josef Albers | Sciencx - » Porting Phi-3-Vision to MLX: A Python Hobbyist’s Journey. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2024/08/07/porting-phi-3-vision-to-mlx-a-python-hobbyists-journey/
CHICAGO
" » Porting Phi-3-Vision to MLX: A Python Hobbyist’s Journey." Josef Albers | Sciencx - Accessed . https://www.scien.cx/2024/08/07/porting-phi-3-vision-to-mlx-a-python-hobbyists-journey/
IEEE
" » Porting Phi-3-Vision to MLX: A Python Hobbyist’s Journey." Josef Albers | Sciencx [Online]. Available: https://www.scien.cx/2024/08/07/porting-phi-3-vision-to-mlx-a-python-hobbyists-journey/. [Accessed: ]
rf:citation
» Porting Phi-3-Vision to MLX: A Python Hobbyist’s Journey | Josef Albers | Sciencx | https://www.scien.cx/2024/08/07/porting-phi-3-vision-to-mlx-a-python-hobbyists-journey/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts