ESP32 Compilation & Upload & Speed

[mnfisher]

An idea...

For 'advanced' esp32 Flowcoders out there... You must have a working esp32 (v5) setup first...

I need to be able to use different config files for esp32 projects - and also add components etc which makes the newest 'single' compilation directory inconvenient.

One option is to rollback the batch files - but I wrote this as an experiment (and never having really worked with batch files before it might be a bit rough)

So - having backed up your current file this replaces C:\ProgramData\MatrixTSL\FlowcodeV10\FCD\ESP\Batch\esp32_build.bat

This reverts to the old style 'directory' per project. It also uses idf.py create-project rather than building the directories from scratch - but maintains compatibility with everything. I did wonder about using the project_name.c file rather than the esp_project.c - but have stayed true to the FC technique (not least to avoid breaking all my old projects)

There is a slight issue - the batch file (and arguments) used seems to be 'fixed' - so I have hard coded the com port (I have COM6 currently) - there is maybe a workaround - where the COM port can be passed as a parameter.

Then - clicking 'compile to hex' builds and uploads the program. It has a speed advantage, especially for slower machines, because it just runs export.bat once. First compilation is slow - it does ~1200 files - but thereafter it is quicker. Also it's much quicker if comms failed (PuTTY open?) - as it doesn't attempt to recompile.

I removed the 'stats' - I don't often look at them - but they could also be added if needed.
Also - it might be useful to add 'git init' - for a little version control..

Any thoughts / improvements / requests?

You'll probably need to change the COM port in the last line (and if you are lucky might get away with a faster baud rate?)

Martin

Code: Select all

REM CRC: 0796FC1E6A776AF9156B257209C35258CE7C28B3EC20AEA2C632DB392A0E4BFD5B3F53B21811978F7B21FE4085045A7E774D1D3D69D8A99EE7320EAE740A5DB69CE852FCC2E76CD08880CFFD18B90BBB3A0DDF873040DD7BD8DA408700E5719B7AED6A0109421CF3B199900461C78937B42DEF8219F63A24580137A2F6C8FDB7553324EFF791947E8AB6FFD612B81BAA97C798A8F1CFC189DE5D38517747C774AF2DD6F839BAECF83F7F994C7164CE77868F480396BDEF2F2C3050DA273BE4EE6448CEF0EFC2793410C8AE2F408667340D60716539085AD7C37F16299AE1E9D4E82474DD52B742AE
REM REVISION: 14.0
REM GUID: 955EA920-4961-4C53-BA6B-EF4AF66BB4C2
@SET BASE_DIR=%~1
@SET TARGET=%~2
@SET OUT_DIR=%~3
@SET CONFIG=%~4
@SET CHIPTYPE=%~5
@ECHO OFF

@SET DOS_BUILD_DIR=%OUT_DIR%\%TARGET%
@CALL %BASE_DIR%export.bat> nul
@REM Remove spaces from the path ...
@SET DOS_BUILD_DIR=%DOS_BUILD_DIR: =_%
@ECHO Building: %TARGET% ...

@REM Create the project tree and copy in default files ...

IF EXIST %OUT_DIR%%TARGET% GOTO CREATED
idf.py create-project %TARGET% --path %OUT_DIR% 
@PUSHD %~dp0
@COPY /Y mainCMakeLists.txt "%DOS_BUILD_DIR%\main\CMakeLists.txt" > nul
@COPY /Y component.mk "%DOS_BUILD_DIR%\main" > nul
@COPY /Y %CONFIG%.txt "%DOS_BUILD_DIR%\sdkconfig" > nul
@POPD
cd %DOS_BUILD_DIR%
@COPY "%OUT_DIR%%TARGET%.c" "%DOS_BUILD_DIR%\main\esp-project.c" > NUL
idf.py set-target %CHIPTYPE%
GOTO BUILD

:CREATED

cd %DOS_BUILD_DIR%
@COPY /Y "%OUT_DIR%%TARGET%.c" "%DOS_BUILD_DIR%\main\esp-project.c" > NUL

:BUILD
idf.py -p COM6 -b 921600 flash

[Steve-Matrix]

Hi Martin, Thanks for this. Hopefully this will be of use to some of our users.

Our original approach was to rebuild everything when the project name was changed, but there were many complaints that this process was far too slow. Hence our change to a common compilation directory. For most users, this new approach will be fine. But we do understand that for more advanced users who are switching between different ESP configurations then it causes issues.

You should be able to have the two systems operate side-by-side by adding a custom compiler setting in compiler options.

You could adopt a hybrid approach that allows fast compilation and multiple config settings. For example, when you have a single-directory config setting you want to reuse, you could rename the whole folder. You will eventually get multiple compilation folders that suit a specific configuration setup. All you then need to do it rename the appropriate folder back to the original single-use folder name. You could automate this within the batch file so it automatically renames the folders for you when you select a specific config type.

BTW, the COM port is passed as a parameter. But it is only passed as a parameter to the "Programmer" batch file, as that is the one that actually sends the compiled project to the device. You can add Flowcode's config setting as a parameter to a custom "Compiler" setting if you like.

[Steve-Matrix]

Also, this might be useful to understand parameters in the batch files:

%1, %2, etc are the parameters passed to the batch file, and %0 is the batch file itself.

• %0 = the full path to this batch file
• %d0 = the drive letter of this batch file
• %p0 = the directory of this batch file
• %dp0 = the drive + directory of this batch file

The “”~” strips the quotes.

There are also some other ways to manipulate this (and other file-based) parameters to the batch file (and indeed any internal variables too)…

Code: Select all

%~I - expands %I removing any surrounding quotes ("")
%~fI - expands %I to a fully qualified path name
%~dI - expands %I to a drive letter only
%~pI - expands %I to a path only
%~nI - expands %I to a file name only
%~xI - expands %I to a file extension only
%~sI - expanded path contains short names only
%~aI - expands %I to file attributes of file
%~tI - expands %I to date/time of file
%~zI - expands %I to size of file
%~$PATH:I - searches the directories listed in the PATH
environment variable and expands %I to the
fully qualified name of the first one found.
If the environment variable name is not
defined or the file is not found by the
search, then this modifier expands to the
empty string

[mnfisher]

Thanks Steve,

I knew the murky world of batch files (I think the 'DOS_' says it all) contains many dark secrets and corners....

It's only ever the first compilation that is slow (or after a config change) when the idf tools rebuilds everything - and this approach is noticeably quicker for recompilation
I almost use a 'test driven development' technique - where I use simulation when possible - but test on the hardware frequently too. (I'd maybe use an Arduino as a test=bed as compilation/upload is so much quicker too though this isn't always possible) - so maybe recompile more than most?

I'd wondered about adding this as a 'custom' batch file (or adding COM port as an argument to the compiler command line), but compiler options has 'Settings can't be changed because they are being overridden by the current target definition file to use these settings change the target device or edit the definition file' so hadn't tried this.

If these weren't (or maybe they aren't?) locked down - then compile and programmer batch files would be almost identical (compile not needing to upload) although only one or the other would need to be called (not both) which would require some changes to the main program.

Martin

[chipfryer27]

Hi

Yesterday was the the second time I used an ESP since installing the toolchain as per latest instructions.

A couple of weeks or so ago I compiled, which took time but not as long as before (which could be down to a better laptop) and things worked without issue. Even any update to the program was noticeably faster compared to updating under previous toolchain. All in all an improvement.

However what surprised me immensly was that yesterday when I compiled an entirely different program (to the same ESP chip) it did it pretty much in the same time as an "update". No compilation of 1200 lines or the like (or if I did it did it without me noticing <s>).

Renaming the folder in the toolchain then caused a "full compilation" to take place. So, as I don't use ESP that much if I want to keep anything I'll just name the folder to perhaps my project name.

The new toolchain ensemble is definitely an improvement.

Regards

[mnfisher]

Can tidy up a bit (get rid of those GOTOs)

By changing the relevant section to

Code: Select all

IF NOT EXIST %DOS_BUILD_DIR% (
   idf.py create-project %TARGET% --path %OUT_DIR% 
   @PUSHD %~dp0
   @COPY /Y mainCMakeLists.txt "%DOS_BUILD_DIR%\main\CMakeLists.txt" > nul
   @COPY /Y component.mk "%DOS_BUILD_DIR%\main" > nul
   @COPY /Y %CONFIG%.txt "%DOS_BUILD_DIR%\sdkconfig" > nul
   @POPD
   cd %DOS_BUILD_DIR%
   @COPY "%OUT_DIR%%TARGET%.c" "%DOS_BUILD_DIR%\main\esp-project.c" > NUL
   idf.py set-target %CHIPTYPE%
)

cd %DOS_BUILD_DIR%
@COPY /Y "%OUT_DIR%%TARGET%.c" "%DOS_BUILD_DIR%\main\esp-project.c" > NUL
idf.py -p COM6 -b 921600 flash

But I also started looking if was possible to not rebuild if nothing has changed and just upload(FC warns)... It always builds (quickly) - because the source file changes (if only the comments)

Using fc (file compare) doesn't work - the comments have changed - using something like "beyond compare" (another
toolkit recommendation) then it would be possible to ignore the comments too

Martin